Nov 02 09:34:36.765640 ip-10-0-134-174 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 09:34:37.220781 ip-10-0-134-174 kubenswrapper[2545]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 09:34:37.220781 ip-10-0-134-174 kubenswrapper[2545]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 09:34:37.220781 ip-10-0-134-174 kubenswrapper[2545]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 09:34:37.221420 ip-10-0-134-174 kubenswrapper[2545]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 09:34:37.224976 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.224889    2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:37.224976 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.224972    2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:37.224976 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.224976    2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:37.224976 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.224979    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:37.224976 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.224982    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.224986    2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.224989    2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.224992    2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.224994    2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.224997    2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225000    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225003    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225005    2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225008    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225010    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225013    2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225015    2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225018    2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225021    2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225023    2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225025    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225028    2545 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225031    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:37.225162 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225033    2545 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225036    2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225038    2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225041    2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225044    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225048    2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225053    2545 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225057    2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225060    2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225063    2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225065    2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225068    2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225070    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225073    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225076    2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225079    2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225082    2545 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225085    2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225088    2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225091    2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:37.225632 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225093    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225096    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225098    2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225101    2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225104    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225106    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225109    2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225112    2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225115    2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225118    2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225120    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225123    2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225126    2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225129    2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225131    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225134    2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225137    2545 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225140    2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225142    2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:37.226098 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225144    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225147    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225150    2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225152    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225155    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225157    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225160    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225163    2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225166    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225169    2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225172    2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225175    2545 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225177    2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225180    2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225182    2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225184    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225189    2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225194    2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225197    2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:37.226604 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225200    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225203    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225207    2545 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225210    2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225213    2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225216    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225218    2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225221    2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225224    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225226    2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225229    2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225231    2545 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225234    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225237    2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225239    2545 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225241    2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225244    2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225246    2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225249    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225252    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:37.227049 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225255    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:37.227525 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225259    2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:37.227525 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225261    2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:37.227525 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225264    2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:37.227525 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.225267    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227535    2545 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227543    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227546    2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227549    2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227552    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227555    2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227558    2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227560    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227563    2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227566    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227569    2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227572    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227574    2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227577    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227579    2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227582    2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227585    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227587    2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227590    2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:37.227659 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227593    2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227595    2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227597    2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227600    2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227603    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227605    2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227608    2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227610    2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227613    2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227616    2545 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227618    2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227621    2545 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227623    2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227626    2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227629    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227632    2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227635    2545 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227637    2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227641    2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227643    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:37.228138 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227646    2545 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227648    2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227651    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227653    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227657    2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227659    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227662    2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227665    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227668    2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227670    2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227673    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227676    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227679    2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227682    2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227685    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227687    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227690    2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227692    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227695    2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:37.228618 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227698    2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227700    2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227702    2545 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227705    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227708    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227710    2545 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227713    2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227717    2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227720    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227723    2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227725    2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227728    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227731    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227734    2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227736    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227738    2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227741    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227744    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227747    2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:37.229073 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227750    2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227754    2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227757    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227761    2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227765    2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227768    2545 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227770    2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227773    2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227775    2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227778    2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227781    2545 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227783    2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227786    2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227788    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227791    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227794    2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227796    2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227799    2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227801    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:37.229535 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227804    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227807    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227810    2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227812    2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227815    2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227817    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227820    2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227823    2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227825    2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.227829    2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228305    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228309    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228313    2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228316    2545 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228318    2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228321    2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228324    2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228327    2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228330    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228332    2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:37.230069 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228335    2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228337    2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228340    2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228343    2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228346    2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228348    2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228367    2545 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228370    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228373    2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228375    2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228378    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228380    2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228383    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228386    2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228389    2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228391    2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228394    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228396    2545 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228399    2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228401    2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:37.230596 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228404    2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228406    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228409    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228411    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228413    2545 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228417    2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228420    2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228422    2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228425    2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228428    2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228431    2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228433    2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228436    2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228438    2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228441    2545 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228445    2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228449    2545 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228452    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228455    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:37.231071 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228458    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228460    2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228463    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228466    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228469    2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228471    2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228474    2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228478    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228480    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228483    2545 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228485    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228488    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228490    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228493    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228496    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228498    2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228501    2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228503    2545 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228506    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228510    2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:37.231528 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228512    2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228515    2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228518    2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228521    2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228524    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228528    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228531    2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228534    2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228536    2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228539    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228542    2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228545    2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228547    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228550    2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228552    2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228555    2545 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228557    2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228560    2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228563    2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:37.232000 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228565    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228567    2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228570    2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228572    2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228575    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228577    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228580    2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228582    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228585    2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228589    2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228591    2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228594    2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228596    2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228599    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228601    2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228606    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228609    2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.228611    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229663    2545 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229673    2545 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 09:34:37.232457 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229680    2545 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229685    2545 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229690    2545 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229693    2545 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229698    2545 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229702    2545 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229706    2545 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229709    2545 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229712    2545 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229716    2545 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229719    2545 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229721    2545 flags.go:64] FLAG: --cgroup-root=""
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229724    2545 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229728    2545 flags.go:64] FLAG: --client-ca-file=""
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229731    2545 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229734    2545 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229738    2545 flags.go:64] FLAG: --cluster-domain=""
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229741    2545 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229744    2545 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229748    2545 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229751    2545 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229760    2545 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229763    2545 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 09:34:37.232921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229766    2545 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229769    2545 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229772    2545 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229776    2545 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229781    2545 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229784    2545 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229788    2545 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229792    2545 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229795    2545 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229798    2545 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229801    2545 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229804    2545 flags.go:64] FLAG: --enable-server="true"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229807    2545 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229812    2545 flags.go:64] FLAG: --event-burst="100"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229815    2545 flags.go:64] FLAG: --event-qps="50"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229818    2545 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229821    2545 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229824    2545 flags.go:64] FLAG: --eviction-hard=""
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229828    2545 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229831    2545 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229834    2545 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229837    2545 flags.go:64] FLAG: --eviction-soft=""
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229840    2545 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229843    2545 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229846    2545 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 09:34:37.233464 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229848    2545 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229851    2545 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229854    2545 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229857    2545 flags.go:64] FLAG: --feature-gates=""
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229861    2545 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229864    2545 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229866    2545 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229870    2545 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229873    2545 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229876    2545 flags.go:64] FLAG: --help="false"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229879    2545 flags.go:64] FLAG: --hostname-override="ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229884    2545 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229887    2545 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229891    2545 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229894    2545 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229897    2545 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229900    2545 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229903    2545 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229906    2545 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229909    2545 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229912    2545 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229915    2545 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229918    2545 flags.go:64] FLAG: --kube-reserved=""
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229920    2545 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 09:34:37.234113 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229923    2545 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229926    2545 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229929    2545 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229932    2545 flags.go:64] FLAG: --lock-file=""
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229935    2545 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229938    2545 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229941    2545 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229946    2545 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229949    2545 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229952    2545 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229955    2545 flags.go:64] FLAG: --logging-format="text"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229958    2545 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229961    2545 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229963    2545 flags.go:64] FLAG: --manifest-url=""
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229966    2545 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229971    2545 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229974    2545 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229977    2545 flags.go:64] FLAG: --max-pods="110"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229980    2545 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229984    2545 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229988    2545 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229991    2545 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229994    2545 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.229997    2545 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230000    2545 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 09:34:37.234713 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230007    2545 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230010    2545 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230013    2545 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230016    2545 flags.go:64] FLAG: --pod-cidr=""
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230019    2545 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230022    2545 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230025    2545 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230028    2545 flags.go:64] FLAG: --port="10250"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230031    2545 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230034    2545 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-07c983f20e53ce506"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230037    2545 flags.go:64] FLAG: --qos-reserved=""
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230040    2545 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230043    2545 flags.go:64] FLAG: --register-node="true"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230046    2545 flags.go:64] FLAG: --register-with-taints=""
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230050    2545 flags.go:64] FLAG: --registry-burst="10"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230052    2545 flags.go:64] FLAG: --registry-qps="5"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230055    2545 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230058    2545 flags.go:64] FLAG: --reserved-memory=""
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230062    2545 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230065    2545 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230068    2545 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230071    2545 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230074    2545 flags.go:64] FLAG: --runonce="false"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230077    2545 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230081    2545 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230084    2545 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 09:34:37.235289 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230087    2545 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230090    2545 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230094    2545 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230097    2545 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230100    2545 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230103    2545 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230106    2545 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230109    2545 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230112    2545 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230115    2545 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230118    2545 flags.go:64] FLAG: --system-cgroups=""
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230121    2545 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230126    2545 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230129    2545 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230132    2545 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230135    2545 flags.go:64] FLAG: --tls-min-version=""
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230138    2545 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230141    2545 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230144    2545 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230147    2545 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230150    2545 flags.go:64] FLAG: --v="2"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230154    2545 flags.go:64] FLAG: --version="false"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230158    2545 flags.go:64] FLAG: --vmodule=""
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230162    2545 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230165    2545 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 09:34:37.235981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230315    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230319    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230322    2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230325    2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230327    2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230330    2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230333    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230335    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230339    2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230342    2545 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230346    2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230361    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230364    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230367    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230370    2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230372    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230375    2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230379    2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230384    2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:37.236588 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230387    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230390    2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230393    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230396    2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230399    2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230401    2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230404    2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230407    2545 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230409    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230412    2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230414    2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230417    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230420    2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230422    2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230425    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230428    2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230431    2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230434    2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230437    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:37.237031 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230440    2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230442    2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230448    2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230450    2545 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230453    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230458    2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230461    2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230463    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230466    2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230468    2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230471    2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230474    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230476    2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230479    2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230482    2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230484    2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230487    2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230489    2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230492    2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230494    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:37.237518 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230497    2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230500    2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230502    2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230505    2545 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230507    2545 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230510    2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230512    2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230515    2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230517    2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230520    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230522    2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230525    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230528    2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230530    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230534    2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230537    2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230540    2545 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230542    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230545    2545 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230549    2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:37.237981 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230553    2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230556    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230559    2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230562    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230565    2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230567    2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230570    2545 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230572    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230575    2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230579    2545 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230582    2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230584    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230587    2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230589    2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230592    2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230594    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230605    2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230609    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230612    2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230615    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:37.238475 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230617    2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230620    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230623    2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230625    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230628    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230631    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230635    2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230637    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.230664    2545 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230700    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230705    2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230708    2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230711    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230714    2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230717    2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:37.238954 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230720    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230722    2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230725    2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230728    2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230731    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230733    2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230737    2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230740    2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230742    2545 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230745    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230748    2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230750    2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230753    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230755    2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230758    2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230761    2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230763    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230766    2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230768    2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230771    2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:37.239341 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230773    2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230776    2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230778    2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230781    2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230784    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230787    2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230790    2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230792    2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230795    2545 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230798    2545 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230801    2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230803    2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230806    2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230808    2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230811    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230813    2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230816    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230819    2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230823    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230825    2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:37.239829 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230828    2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230831    2545 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230833    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230836    2545 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230840    2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230844    2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230847    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230849    2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230852    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230854    2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230857    2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230860    2545 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230862    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230865    2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230867    2545 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230870    2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230874    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230876    2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230879    2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230881    2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:37.240293 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230884    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230886    2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230889    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230892    2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230894    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230897    2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230899    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230902    2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230905    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230907    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230911    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230914    2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230916    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230919    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230921    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230924    2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230927    2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230929    2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230932    2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:37.240772 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230934    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230937    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230939    2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230942    2545 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230945    2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230947    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230950    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230952    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230955    2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230958    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230961    2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230965    2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230968    2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230971    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230974    2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230976    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230980    2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230983    2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230985    2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:37.241219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230988    2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.230990    2545 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.237451    2545 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.237466    2545 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.237483    2545 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.237488    2545 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237543    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237549    2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237552    2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237555    2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237558    2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237561    2545 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237563    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237566    2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237568    2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237571    2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237574    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237576    2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237579    2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237582    2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:37.241694 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237584    2545 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237587    2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237590    2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237592    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237595    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237598    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237600    2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237603    2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237606    2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237608    2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237611    2545 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237614    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237616    2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237619    2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237622    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237626    2545 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237630    2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237634    2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237637    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:37.242164 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237640    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237642    2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237645    2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237648    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237650    2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237653    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237655    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237658    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237661    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237663    2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237666    2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237668    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237671    2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237674    2545 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237677    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237679    2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237682    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237685    2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237688    2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:37.242617 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237690    2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237693    2545 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237696    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237698    2545 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237701    2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237704    2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237706    2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237709    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237711    2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237714    2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237719    2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237723    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237726    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237728    2545 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237731    2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237734    2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237736    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237739    2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237741    2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:37.243045 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237744    2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237747    2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237749    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237752    2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237754    2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237757    2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237759    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237762    2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237765    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237767    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237770    2545 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237773    2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237775    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237778    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237780    2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237783    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237785    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237788    2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237791    2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237793    2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:37.243501 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237796    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237799    2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237802    2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237804    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237807    2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237810    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237812    2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237815    2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237817    2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237820    2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237822    2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237825    2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237827    2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237830    2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237832    2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.237860    2545 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 09:34:37.243975 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.237996    2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238000    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238003    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238007    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238010    2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238012    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238015    2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238018    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238020    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238023    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238025    2545 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238028    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238030    2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238033    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238035    2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238038    2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238041    2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238043    2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238046    2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238048    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:37.244709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238051    2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238054    2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238057    2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238060    2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238062    2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238065    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238067    2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238070    2545 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238072    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238075    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238077    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238080    2545 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238082    2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238085    2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238087    2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238090    2545 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238092    2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238095    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238097    2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238100    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:37.245221 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238103    2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238105    2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238108    2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238110    2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238113    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238116    2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238118    2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238121    2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238123    2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238126    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238129    2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238131    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238134    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238136    2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238139    2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238142    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238144    2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238147    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238149    2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:37.245808 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238152    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238154    2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238156    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238159    2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238161    2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238165    2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238169    2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238172    2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238175    2545 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238177    2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238180    2545 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238183    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238185    2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238188    2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238191    2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238194    2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238198    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238202    2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238221    2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:37.246561 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238225    2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238228    2545 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238232    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238234    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238240    2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238242    2545 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238245    2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238248    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238250    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238253    2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238256    2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238258    2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238261    2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238263    2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238266    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238269    2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238271    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238274    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238276    2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238279    2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:37.247128 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238281    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238283    2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238286    2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238288    2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238291    2545 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238293    2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238296    2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238299    2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.238320    2545 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238363    2545 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238367    2545 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238371    2545 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238374    2545 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238376    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238379    2545 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:34:37.247619 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238381    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238386    2545 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238388    2545 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238391    2545 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238394    2545 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238396    2545 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238399    2545 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238402    2545 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238405    2545 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238408    2545 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238410    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238413    2545 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238415    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238418    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238420    2545 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238423    2545 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238426    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238428    2545 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238431    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:34:37.247968 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238433    2545 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238436    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238438    2545 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238441    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238444    2545 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238447    2545 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238451    2545 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238454    2545 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238458    2545 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238461    2545 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238464    2545 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238467    2545 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238470    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238473    2545 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238477    2545 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238480    2545 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238484    2545 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238488    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238491    2545 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:34:37.248421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238494    2545 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238496    2545 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238499    2545 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238502    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238505    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238507    2545 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238510    2545 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238512    2545 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238515    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238517    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238520    2545 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238522    2545 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238525    2545 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238528    2545 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238530    2545 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238533    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238535    2545 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238538    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238541    2545 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238543    2545 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:34:37.248855 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238546    2545 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238548    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238551    2545 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238567    2545 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238570    2545 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238573    2545 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238575    2545 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238579    2545 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238582    2545 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238584    2545 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238587    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238589    2545 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238592    2545 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238595    2545 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238604    2545 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238609    2545 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238612    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238614    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238617    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238619    2545 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:34:37.249319 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238622    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238624    2545 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238627    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238629    2545 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238634    2545 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238636    2545 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238639    2545 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238642    2545 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238644    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238647    2545 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238650    2545 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238652    2545 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238655    2545 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238657    2545 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238660    2545 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238662    2545 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238665    2545 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238667    2545 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238670    2545 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238674    2545 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:34:37.249869 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238677    2545 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:34:37.250333 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:37.238680    2545 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:34:37.250333 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.238731    2545 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 09:34:37.250333 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.240720    2545 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 09:34:37.250333 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.241903    2545 server.go:1014] "Starting client certificate rotation"
Nov 02 09:34:37.250333 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.242003    2545 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 09:34:37.250333 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.242042    2545 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 09:34:37.250333 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.244043    2545 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 09:34:37.250333 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.249029    2545 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 09:34:37.260888 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.260874    2545 log.go:25] "Validated CRI v1 runtime API"
Nov 02 09:34:37.266243 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.266224    2545 log.go:25] "Validated CRI v1 image API"
Nov 02 09:34:37.267085 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.267062    2545 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 09:34:37.269203 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.269174    2545 fs.go:135] Filesystem UUIDs: map[2024e37c-a86a-4ede-8331-96dbd215a633:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c503b405-9dea-45a0-86db-375ddf86232d:/dev/nvme0n1p3]
Nov 02 09:34:37.269276 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.269200    2545 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 09:34:37.273399 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.273277    2545 manager.go:217] Machine: {Timestamp:2026-11-02 09:34:37.272822343 +0000 UTC m=+0.384605714 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101001 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f158347127cc5ae4467fed90ef1ee SystemUUID:ec2f1583-4712-7cc5-ae44-67fed90ef1ee BootID:8da1a5d0-6c11-4a2e-a6d1-64b974c4e3ea Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:66:2a:01 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:66:2a:01 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:25:31:7c:af:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 09:34:37.273399 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.273392    2545 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 09:34:37.273518 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.273465    2545 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 09:34:37.276940 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.276920    2545 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 09:34:37.277196 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.277175    2545 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 09:34:37.277371 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.277197    2545 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-174.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 09:34:37.277424 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.277374    2545 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 09:34:37.277424 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.277383    2545 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 09:34:37.277424 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.277411    2545 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 09:34:37.277501 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.277427    2545 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 09:34:37.278269 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.278259    2545 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 09:34:37.282152 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.282140    2545 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 09:34:37.282266 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.282256    2545 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 09:34:37.282303 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.282287    2545 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 09:34:37.282332 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.282310    2545 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 09:34:37.282332 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.282321    2545 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 09:34:37.282332 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.282329    2545 kubelet.go:405] "Adding apiserver pod source"
Nov 02 09:34:37.282468 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.282338    2545 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 09:34:37.282686 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.282675    2545 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 09:34:37.284921 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.284906    2545 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 09:34:37.290780 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.290763    2545 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 09:34:37.291552 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.291540    2545 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 09:34:37.291606 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.291570    2545 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 09:34:37.291727 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.291717    2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 09:34:37.291759 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.291733    2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 09:34:37.291759 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.291740    2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 09:34:37.291759 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.291745    2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 09:34:37.291759 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.291751    2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 09:34:37.291759 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.291757    2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 09:34:37.292396 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.292388    2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 09:34:37.292426 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.292397    2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 09:34:37.292426 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.292405    2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 09:34:37.292426 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.292411    2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 09:34:37.292426 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.292419    2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 09:34:37.292426 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.292428    2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 09:34:37.293976 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.293960    2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 09:34:37.293976 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.293971    2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 09:34:37.297666 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.297651    2545 server.go:1263] "Started kubelet"
Nov 02 09:34:37.297783 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.297715    2545 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 09:34:37.297837 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.297750    2545 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 09:34:37.297837 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.297823    2545 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 09:34:37.298415 ip-10-0-134-174 systemd[1]: Started Kubernetes Kubelet.
Nov 02 09:34:37.299451 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.298684    2545 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-174.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 09:34:37.299451 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.298771    2545 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 09:34:37.301114 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.301097    2545 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 09:34:37.303821 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.303789    2545 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 09:34:37.305227 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.305206    2545 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 09:34:37.306001 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.305986    2545 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 09:34:37.306108 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.306088    2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-174.us-east-2.compute.internal\" not found"
Nov 02 09:34:37.306870 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.306851    2545 factory.go:55] Registering systemd factory
Nov 02 09:34:37.306870 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.306863    2545 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 09:34:37.306982 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.306875    2545 factory.go:223] Registration of the systemd container factory successfully
Nov 02 09:34:37.306982 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.306857    2545 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 09:34:37.306982 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.306926    2545 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 09:34:37.306982 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.306966    2545 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 09:34:37.306982 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.306973    2545 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 09:34:37.307192 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.307101    2545 factory.go:153] Registering CRI-O factory
Nov 02 09:34:37.307192 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.307114    2545 factory.go:223] Registration of the crio container factory successfully
Nov 02 09:34:37.307192 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.307162    2545 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 09:34:37.307192 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.307184    2545 factory.go:103] Registering Raw factory
Nov 02 09:34:37.307334 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.307207    2545 manager.go:1199] Started watching for new ooms in manager
Nov 02 09:34:37.307957 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.307943    2545 manager.go:319] Starting recovery of all containers
Nov 02 09:34:37.308438 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.308405    2545 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-174.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 09:34:37.308576 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.308555    2545 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 09:34:37.308831 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.307802    2545 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-174.us-east-2.compute.internal.18e430ae6c2cd6a4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-174.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-174.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 09:34:37.297620644 +0000 UTC m=+0.409404015,LastTimestamp:2026-11-02 09:34:37.297620644 +0000 UTC m=+0.409404015,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-174.us-east-2.compute.internal,}"
Nov 02 09:34:37.314934 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.314902    2545 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-174.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 09:34:37.319374 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.319339    2545 manager.go:324] Recovery completed
Nov 02 09:34:37.320612 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.320594    2545 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 09:34:37.324826 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.324813    2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:34:37.327397 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.327373    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:34:37.327482 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.327408    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:34:37.327482 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.327422    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:34:37.328009 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.327996    2545 cpu_manager.go:226] "Starting" policy="none"
Nov 02 09:34:37.328009 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.328007    2545 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 09:34:37.328125 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.328010    2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:34:37.328125 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.328025    2545 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 09:34:37.328755 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.328735    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:34:37.328837 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.328773    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:34:37.328837 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.328790    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:34:37.329807 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.329715    2545 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-174.us-east-2.compute.internal.18e430ae6df31e37  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-174.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-174.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-174.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 09:34:37.327392311 +0000 UTC m=+0.439175684,LastTimestamp:2026-11-02 09:34:37.327392311 +0000 UTC m=+0.439175684,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-174.us-east-2.compute.internal,}"
Nov 02 09:34:37.330298 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.330286    2545 policy_none.go:50] "Start"
Nov 02 09:34:37.330347 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.330301    2545 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 09:34:37.330347 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.330311    2545 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 09:34:37.332458 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.332446    2545 policy_none.go:44] "Start"
Nov 02 09:34:37.333496 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.333480    2545 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bg6nv"
Nov 02 09:34:37.341622 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.341604    2545 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bg6nv"
Nov 02 09:34:37.342187 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.342119    2545 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-174.us-east-2.compute.internal.18e430ae6df3749d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-174.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-174.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-174.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 09:34:37.327414429 +0000 UTC m=+0.439197804,LastTimestamp:2026-11-02 09:34:37.327414429 +0000 UTC m=+0.439197804,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-174.us-east-2.compute.internal,}"
Nov 02 09:34:37.355902 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.355875    2545 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 09:34:37.358203 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.358188    2545 manager.go:341] "Starting Device Plugin manager"
Nov 02 09:34:37.358299 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.358219    2545 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 09:34:37.358299 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.358229    2545 server.go:87] "Starting device plugin registration server"
Nov 02 09:34:37.358470 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.358458    2545 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 09:34:37.358527 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.358471    2545 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 09:34:37.358668 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.358620    2545 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 09:34:37.358728 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.358694    2545 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 09:34:37.358728 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.358703    2545 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 09:34:37.359079 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.359047    2545 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 09:34:37.359152 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.359091    2545 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-174.us-east-2.compute.internal\" not found"
Nov 02 09:34:37.434202 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.434168    2545 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 09:34:37.436063 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.435295    2545 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 09:34:37.436063 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.435315    2545 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 09:34:37.436063 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.435344    2545 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 09:34:37.436063 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.435391    2545 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 09:34:37.439451 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.439431    2545 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:34:37.459471 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.459444    2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:34:37.460670 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.460655    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:34:37.460759 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.460683    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:34:37.460759 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.460696    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:34:37.460759 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.460720    2545 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.472613 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.472561    2545 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.472613 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.472588    2545 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-174.us-east-2.compute.internal\": node \"ip-10-0-134-174.us-east-2.compute.internal\" not found"
Nov 02 09:34:37.493125 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.493100    2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-174.us-east-2.compute.internal\" not found"
Nov 02 09:34:37.536100 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.536075    2545 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"]
Nov 02 09:34:37.536202 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.536130    2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:34:37.537013 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.536997    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:34:37.537101 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.537029    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:34:37.537101 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.537044    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:34:37.539254 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.539239    2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:34:37.539417 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.539402    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.539467 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.539452    2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:34:37.539983 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.539966    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:34:37.540038 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.539964    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:34:37.540038 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.540016    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:34:37.540038 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.540026    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:34:37.540122 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.539994    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:34:37.540122 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.540071    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:34:37.542373 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.542344    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.542458 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.542386    2545 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:34:37.543130 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.543108    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:34:37.543209 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.543147    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:34:37.543209 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.543162    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:34:37.560572 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.560558    2545 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-174.us-east-2.compute.internal\" not found" node="ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.562639 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.562625    2545 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-174.us-east-2.compute.internal\" not found" node="ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.593386 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.593364    2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-174.us-east-2.compute.internal\" not found"
Nov 02 09:34:37.609023 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.609005    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33ba6db318717035e56c8b1001fb10e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"33ba6db318717035e56c8b1001fb10e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.609095 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.609028    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33ba6db318717035e56c8b1001fb10e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"33ba6db318717035e56c8b1001fb10e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.609095 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.609045    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2d480d50be16b70f16fe76e324fa19d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"a2d480d50be16b70f16fe76e324fa19d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.694414 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.694376    2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-174.us-east-2.compute.internal\" not found"
Nov 02 09:34:37.709801 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.709775    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33ba6db318717035e56c8b1001fb10e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"33ba6db318717035e56c8b1001fb10e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.709871 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.709854    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33ba6db318717035e56c8b1001fb10e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"33ba6db318717035e56c8b1001fb10e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.709914 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.709863    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33ba6db318717035e56c8b1001fb10e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"33ba6db318717035e56c8b1001fb10e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.709914 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.709893    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33ba6db318717035e56c8b1001fb10e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"33ba6db318717035e56c8b1001fb10e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.709914 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.709906    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2d480d50be16b70f16fe76e324fa19d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"a2d480d50be16b70f16fe76e324fa19d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.710015 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.709939    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2d480d50be16b70f16fe76e324fa19d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"a2d480d50be16b70f16fe76e324fa19d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.795261 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.795186    2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-174.us-east-2.compute.internal\" not found"
Nov 02 09:34:37.863681 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.863659    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.865819 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:37.865803    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:37.896365 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.896232    2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-174.us-east-2.compute.internal\" not found"
Nov 02 09:34:37.996888 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:37.996854    2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-174.us-east-2.compute.internal\" not found"
Nov 02 09:34:38.097428 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:38.097347    2545 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-174.us-east-2.compute.internal\" not found"
Nov 02 09:34:38.184958 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.184931    2545 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:34:38.206948 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.206925    2545 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:38.222229 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.222213    2545 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 09:34:38.224559 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.224543    2545 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:34:38.234152 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.234135    2545 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 09:34:38.241239 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.241223    2545 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 09:34:38.241334 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.241318    2545 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 09:34:38.241389 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.241374    2545 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 09:34:38.282487 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.282460    2545 apiserver.go:52] "Watching apiserver"
Nov 02 09:34:38.301038 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.301017    2545 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 09:34:38.301329 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.301314    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs","openshift-dns/node-resolver-zz4q2","openshift-image-registry/node-ca-xz8m5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-kmqdf","openshift-multus/multus-zhmvb","openshift-multus/network-metrics-daemon-v9xng","openshift-network-diagnostics/network-check-target-h5bzx","kube-system/global-pull-secret-syncer-lxn8z","kube-system/konnectivity-agent-xzrll","kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-gfgcv","openshift-network-operator/iptables-alerter-h6h6n","openshift-ovn-kubernetes/ovnkube-node-wtrqd"]
Nov 02 09:34:38.303708 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.303690    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.305411 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.305393    2545 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 09:34:38.305781 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.305766    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.307659 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.307638    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:34:38.307786 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.307763    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 09:34:38.307936 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.307918    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-982vl\""
Nov 02 09:34:38.307990 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.307941    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:34:38.309477 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.309460    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 09:34:38.309560 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.309484    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 09:34:38.309560 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.309519    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 09:34:38.309560 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.309536    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 09:34:38.309560 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.309540    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 09:34:38.309762 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.309580    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qbp4s\""
Nov 02 09:34:38.309762 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.309682    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 09:34:38.310175 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.310160    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.312516 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.312500    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.312516 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.312507    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-kubelet\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.312655 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.312538    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-systemd\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.312655 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.312566    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-cni-bin\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.312655 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.312592    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.312655 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.312641    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-run\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.312810 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.312688    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-lib-modules\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.312810 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.312784    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-var-lib-kubelet\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.312872 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.312827    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wmsb\" (UniqueName: \"kubernetes.io/projected/adfc6ef2-c0db-4fa7-9d91-95df203dff49-kube-api-access-7wmsb\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.312906 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.312866    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-etc-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.312946 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.312913    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-log-socket\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.312981 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.312956    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/77530df7-545e-42b0-ad16-ce934cb10dab-serviceca\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:34:38.313018 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.312989    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-run-netns\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.313096 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313027    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-var-lib-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.313096 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313047    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.313096 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313080    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-node-log\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.313223 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313109    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-ovnkube-script-lib\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.313223 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313127    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-systemd\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.313223 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313176    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfc6ef2-c0db-4fa7-9d91-95df203dff49-tmp\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.313338 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313228    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-ovn\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.313338 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313257    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77530df7-545e-42b0-ad16-ce934cb10dab-host\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:34:38.313338 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313274    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-modprobe-d\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.313338 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313294    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysconfig\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.313338 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313311    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysctl-d\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.313559 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313371    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-tuned\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.313559 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313403    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-systemd-units\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.313559 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313444    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-ovnkube-config\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.313559 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313477    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqvqm\" (UniqueName: \"kubernetes.io/projected/77530df7-545e-42b0-ad16-ce934cb10dab-kube-api-access-pqvqm\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:34:38.313559 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313516    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-kubernetes\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.313559 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313542    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-slash\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.313739 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313566    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.313739 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313583    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-env-overrides\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.313739 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313607    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-sys\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.313739 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313634    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-host\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.313739 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313658    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/118b62a2-be2c-4119-87d4-914839ecc25a-ovn-node-metrics-cert\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.313739 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313686    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwtd7\" (UniqueName: \"kubernetes.io/projected/118b62a2-be2c-4119-87d4-914839ecc25a-kube-api-access-xwtd7\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.313739 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313719    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-cni-netd\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.313940 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.313751    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysctl-conf\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.317266 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.315230    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.318108 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.318087    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:38.318203 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:38.318161    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:34:38.318841 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.318819    2545 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 09:34:38.320510 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.320494    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:34:38.322951 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.322936    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:34:38.325251 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.325238    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:38.325304 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:38.325289    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:34:38.327584 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.327570    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:38.327628 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:38.327618    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:34:38.329928 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.329908    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:34:38.346442 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.346421    2545 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xth95"
Nov 02 09:34:38.347537 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.347463    2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 09:29:37 +0000 UTC" deadline="2028-04-24 23:51:39.764683908 +0000 UTC"
Nov 02 09:34:38.347537 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.347506    2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12950h17m1.417179752s"
Nov 02 09:34:38.356024 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.356003    2545 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xth95"
Nov 02 09:34:38.408137 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.408114    2545 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 09:34:38.414128 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414101    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysconfig\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.414328 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414139    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-tuned\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.414328 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414166    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-systemd-units\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.414328 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414185    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysconfig\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.414328 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414191    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-ovnkube-config\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.414328 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414220    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-registration-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.414328 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414244    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-systemd-units\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.414328 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414264    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-system-cni-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.414704 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414347    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-etc-kubernetes\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.414704 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414385    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.414704 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414417    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-env-overrides\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.414704 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414438    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-cni-multus\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.414704 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414476    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-daemon-config\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.414704 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414477    2545 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 09:34:38.414704 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414495    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-multus-certs\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.414704 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414529    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9ecca6d-3d39-4f10-9129-06feb4b292fa-iptables-alerter-script\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:34:38.414704 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414556    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-sys\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.414704 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414587    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-host\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.414704 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414645    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-sys\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.414704 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414675    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/118b62a2-be2c-4119-87d4-914839ecc25a-ovn-node-metrics-cert\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.414704 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414695    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-host\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.415365 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414722    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwtd7\" (UniqueName: \"kubernetes.io/projected/118b62a2-be2c-4119-87d4-914839ecc25a-kube-api-access-xwtd7\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.415365 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414740    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-socket-dir-parent\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.415365 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414801    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-kubelet\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.415365 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414833    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.415365 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414850    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfscl\" (UniqueName: \"kubernetes.io/projected/d9ecca6d-3d39-4f10-9129-06feb4b292fa-kube-api-access-nfscl\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:34:38.415365 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414912    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-ovnkube-config\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.415365 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.414955    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-env-overrides\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.415365 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.415118    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-cni-netd\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.415365 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.415157    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-k8s-cni-cncf-io\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.415365 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.415202    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:38.416002 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.415222    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.416002 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.415785    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysctl-conf\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.416002 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.415803    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-kubelet\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.416002 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.415812    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-cni-netd\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.416002 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.415822    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.416002 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.415874    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-kubelet\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.416002 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.415934    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.416002 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.415957    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz5dm\" (UniqueName: \"kubernetes.io/projected/09b69cf9-00f2-48f9-9c66-f22027a2376c-kube-api-access-xz5dm\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.416002 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.415985    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.416002 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416004    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysctl-conf\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416072    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6qw7\" (UniqueName: \"kubernetes.io/projected/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-kube-api-access-l6qw7\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416118    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-cnibin\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416146    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-lib-modules\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416174    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wmsb\" (UniqueName: \"kubernetes.io/projected/adfc6ef2-c0db-4fa7-9d91-95df203dff49-kube-api-access-7wmsb\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416199    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-etc-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416237    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-tmp-dir\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416265    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416292    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfmg8\" (UniqueName: \"kubernetes.io/projected/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-kube-api-access-cfmg8\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416306    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-etc-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416318    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cnibin\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416376    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-cni-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416408    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-cni-bin\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416429    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-hostroot\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416451    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5b95e7b-2ce0-45a1-9859-0d2378576064-dbus\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416486    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-lib-modules\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416500    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-systemd\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.416785 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416532    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfc6ef2-c0db-4fa7-9d91-95df203dff49-tmp\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416572    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77530df7-545e-42b0-ad16-ce934cb10dab-host\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416573    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-systemd\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416619    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5b95e7b-2ce0-45a1-9859-0d2378576064-kubelet-config\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416650    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77530df7-545e-42b0-ad16-ce934cb10dab-host\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416695    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-modprobe-d\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416720    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysctl-d\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416807    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-modprobe-d\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416842    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqvqm\" (UniqueName: \"kubernetes.io/projected/77530df7-545e-42b0-ad16-ce934cb10dab-kube-api-access-pqvqm\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416870    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbxf5\" (UniqueName: \"kubernetes.io/projected/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-kube-api-access-bbxf5\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416914    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-sys-fs\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416939    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-conf-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.416963    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e8df50bb-8d26-42c1-b496-c9d9f56785d9-konnectivity-ca\") pod \"konnectivity-agent-xzrll\" (UID: \"e8df50bb-8d26-42c1-b496-c9d9f56785d9\") " pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.417017    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.417061    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zc5hs\""
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.417090    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.417130    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.417185    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.417188    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 09:34:38.417744 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.417318    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 09:34:38.418621 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.418512    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 09:34:38.419114 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.419087    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-tuned\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.419209 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.419171    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-llzhk\""
Nov 02 09:34:38.419209 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.417062    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysctl-d\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.419334 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.419232    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/118b62a2-be2c-4119-87d4-914839ecc25a-ovn-node-metrics-cert\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.419562 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.419533    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 09:34:38.419669 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.419642    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f2j22\""
Nov 02 09:34:38.419728 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.419703    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7vqld\""
Nov 02 09:34:38.420043 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.420023    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 09:34:38.420232 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.420205    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9tqmn\""
Nov 02 09:34:38.420466 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.420419    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t9b8s\""
Nov 02 09:34:38.420575 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.420512    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfc6ef2-c0db-4fa7-9d91-95df203dff49-tmp\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.420929 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.420756    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-kubernetes\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.422694 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.417321    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-kubernetes\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.422811 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.422725    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-slash\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.422811 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.422768    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.422811 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.422804    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-hosts-file\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:34:38.422994 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.422965    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-device-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.423103 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.422987    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.423103 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423024    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zv58\" (UniqueName: \"kubernetes.io/projected/961f499a-2ed3-4e0d-b5d2-9194d18b696a-kube-api-access-2zv58\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:38.423103 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423066    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-systemd\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.423103 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423087    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 09:34:38.423103 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423100    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-cni-bin\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423089    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-slash\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423131    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423136    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-system-cni-dir\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423149    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423189    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfq4v\" (UniqueName: \"kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v\") pod \"network-check-target-h5bzx\" (UID: \"dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e\") " pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423223    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423226    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-run\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423258    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423264    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-var-lib-kubelet\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423295    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kdt4x\""
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423298    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-log-socket\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423326    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423332    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/77530df7-545e-42b0-ad16-ce934cb10dab-serviceca\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423385    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-netns\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.423400 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423396    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423443    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9ecca6d-3d39-4f10-9129-06feb4b292fa-host-slash\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423451    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423448    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423480    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-run-netns\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423512    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-var-lib-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423514    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-cni-bin\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423546    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423586    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-node-log\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423659    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423676    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423683    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-run-netns\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423708    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-log-socket\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423717    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-var-lib-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423762    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-systemd\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423763    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-var-lib-kubelet\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423764    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-ovnkube-script-lib\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423879    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-run\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423916    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.424722 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.423924    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-node-log\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.425893 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.424463    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:38.425893 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.424511    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-ovn\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.425893 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.424610    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-socket-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.425893 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.424614    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-ovn\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.425893 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.424671    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/77530df7-545e-42b0-ad16-ce934cb10dab-serviceca\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:34:38.425893 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.424710    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09b69cf9-00f2-48f9-9c66-f22027a2376c-cni-binary-copy\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.425893 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.424786    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e8df50bb-8d26-42c1-b496-c9d9f56785d9-agent-certs\") pod \"konnectivity-agent-xzrll\" (UID: \"e8df50bb-8d26-42c1-b496-c9d9f56785d9\") " pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:34:38.425893 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.425141    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-ovnkube-script-lib\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.431558 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.431529    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wmsb\" (UniqueName: \"kubernetes.io/projected/adfc6ef2-c0db-4fa7-9d91-95df203dff49-kube-api-access-7wmsb\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.432922 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.432900    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwtd7\" (UniqueName: \"kubernetes.io/projected/118b62a2-be2c-4119-87d4-914839ecc25a-kube-api-access-xwtd7\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.435051 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.435027    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqvqm\" (UniqueName: \"kubernetes.io/projected/77530df7-545e-42b0-ad16-ce934cb10dab-kube-api-access-pqvqm\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:34:38.525672 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.525646    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfscl\" (UniqueName: \"kubernetes.io/projected/d9ecca6d-3d39-4f10-9129-06feb4b292fa-kube-api-access-nfscl\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:34:38.525672 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.525677    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-k8s-cni-cncf-io\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.525887 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.525694    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:38.525887 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.525711    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.525887 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:38.525823    2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:38.526009 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.525862    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-k8s-cni-cncf-io\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.526009 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:38.525898    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs podName:961f499a-2ed3-4e0d-b5d2-9194d18b696a nodeName:}" failed. No retries permitted until 2026-11-02 09:34:39.025880355 +0000 UTC m=+2.137663726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs") pod "network-metrics-daemon-v9xng" (UID: "961f499a-2ed3-4e0d-b5d2-9194d18b696a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:38.526115 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526032    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz5dm\" (UniqueName: \"kubernetes.io/projected/09b69cf9-00f2-48f9-9c66-f22027a2376c-kube-api-access-xz5dm\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.526115 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526082    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.526252 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526110    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6qw7\" (UniqueName: \"kubernetes.io/projected/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-kube-api-access-l6qw7\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.526252 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526152    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-cnibin\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.526252 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526184    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-tmp-dir\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:34:38.526252 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526201    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.526252 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526219    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfmg8\" (UniqueName: \"kubernetes.io/projected/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-kube-api-access-cfmg8\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.526538 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526272    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cnibin\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.526538 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526271    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.526538 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526288    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.526538 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526386    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.526538 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526440    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-cni-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.526538 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526465    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-cnibin\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.526538 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526472    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-cni-bin\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.526538 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526511    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-cni-bin\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.526538 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526511    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cnibin\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526555    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-hostroot\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526597    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5b95e7b-2ce0-45a1-9859-0d2378576064-dbus\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526604    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-hostroot\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526632    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-cni-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526648    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5b95e7b-2ce0-45a1-9859-0d2378576064-dbus\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526653    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5b95e7b-2ce0-45a1-9859-0d2378576064-kubelet-config\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526687    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5b95e7b-2ce0-45a1-9859-0d2378576064-kubelet-config\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526691    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbxf5\" (UniqueName: \"kubernetes.io/projected/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-kube-api-access-bbxf5\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526721    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-sys-fs\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526747    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-conf-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526771    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e8df50bb-8d26-42c1-b496-c9d9f56785d9-konnectivity-ca\") pod \"konnectivity-agent-xzrll\" (UID: \"e8df50bb-8d26-42c1-b496-c9d9f56785d9\") " pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526805    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-hosts-file\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526832    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-device-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526844    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-sys-fs\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526853    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-conf-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526891    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-hosts-file\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:34:38.526890 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526859    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zv58\" (UniqueName: \"kubernetes.io/projected/961f499a-2ed3-4e0d-b5d2-9194d18b696a-kube-api-access-2zv58\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526914    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-device-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526927    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-system-cni-dir\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526953    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfq4v\" (UniqueName: \"kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v\") pod \"network-check-target-h5bzx\" (UID: \"dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e\") " pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526976    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-system-cni-dir\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.526986    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-netns\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527013    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9ecca6d-3d39-4f10-9129-06feb4b292fa-host-slash\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527043    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527068    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527094    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-socket-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527113    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-netns\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527119    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09b69cf9-00f2-48f9-9c66-f22027a2376c-cni-binary-copy\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527135    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527154    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e8df50bb-8d26-42c1-b496-c9d9f56785d9-agent-certs\") pod \"konnectivity-agent-xzrll\" (UID: \"e8df50bb-8d26-42c1-b496-c9d9f56785d9\") " pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527094    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9ecca6d-3d39-4f10-9129-06feb4b292fa-host-slash\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527207    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-registration-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:38.527241    2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527245    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-system-cni-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.527705 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527251    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-socket-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:38.527290    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret podName:c5b95e7b-2ce0-45a1-9859-0d2378576064 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:39.027273934 +0000 UTC m=+2.139057304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret") pod "global-pull-secret-syncer-lxn8z" (UID: "c5b95e7b-2ce0-45a1-9859-0d2378576064") : object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527277    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-etc-kubernetes\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527317    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-system-cni-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527325    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-registration-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527336    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527343    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-etc-kubernetes\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527378    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-cni-multus\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527442    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-daemon-config\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527472    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-multus-certs\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527481    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-cni-multus\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527499    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9ecca6d-3d39-4f10-9129-06feb4b292fa-iptables-alerter-script\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527521    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-multus-certs\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527536    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-socket-dir-parent\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527563    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-kubelet\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527588    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.528597 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527607    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-socket-dir-parent\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.529415 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.527640    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-kubelet\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.529415 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.528032    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e8df50bb-8d26-42c1-b496-c9d9f56785d9-konnectivity-ca\") pod \"konnectivity-agent-xzrll\" (UID: \"e8df50bb-8d26-42c1-b496-c9d9f56785d9\") " pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:34:38.529415 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.528054    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-tmp-dir\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:34:38.529415 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.528194    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.529415 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.528220    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09b69cf9-00f2-48f9-9c66-f22027a2376c-cni-binary-copy\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.529415 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.528313    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-daemon-config\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.529415 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.528598    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9ecca6d-3d39-4f10-9129-06feb4b292fa-iptables-alerter-script\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:34:38.529847 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.529461    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.531523 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.531501    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e8df50bb-8d26-42c1-b496-c9d9f56785d9-agent-certs\") pod \"konnectivity-agent-xzrll\" (UID: \"e8df50bb-8d26-42c1-b496-c9d9f56785d9\") " pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:34:38.539204 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:38.538044    2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 09:34:38.539204 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:38.538072    2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 09:34:38.539204 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:38.538088    2545 projected.go:196] Error preparing data for projected volume kube-api-access-jfq4v for pod openshift-network-diagnostics/network-check-target-h5bzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:38.539204 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:38.538141    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v podName:dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e nodeName:}" failed. No retries permitted until 2026-11-02 09:34:39.038125476 +0000 UTC m=+2.149908857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jfq4v" (UniqueName: "kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v") pod "network-check-target-h5bzx" (UID: "dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:38.539803 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.539777    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfscl\" (UniqueName: \"kubernetes.io/projected/d9ecca6d-3d39-4f10-9129-06feb4b292fa-kube-api-access-nfscl\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:34:38.540028 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.540006    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz5dm\" (UniqueName: \"kubernetes.io/projected/09b69cf9-00f2-48f9-9c66-f22027a2376c-kube-api-access-xz5dm\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.540327 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.540307    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zv58\" (UniqueName: \"kubernetes.io/projected/961f499a-2ed3-4e0d-b5d2-9194d18b696a-kube-api-access-2zv58\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:38.540480 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.540453    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6qw7\" (UniqueName: \"kubernetes.io/projected/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-kube-api-access-l6qw7\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.540480 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.540472    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfmg8\" (UniqueName: \"kubernetes.io/projected/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-kube-api-access-cfmg8\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.540480 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.540481    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbxf5\" (UniqueName: \"kubernetes.io/projected/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-kube-api-access-bbxf5\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:34:38.551654 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:38.551631    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d480d50be16b70f16fe76e324fa19d.slice/crio-5378d6236e7ea1b301c2da5f0cd98cc06abd991f2cb580cba7d08b14abc5d3cc WatchSource:0}: Error finding container 5378d6236e7ea1b301c2da5f0cd98cc06abd991f2cb580cba7d08b14abc5d3cc: Status 404 returned error can't find the container with id 5378d6236e7ea1b301c2da5f0cd98cc06abd991f2cb580cba7d08b14abc5d3cc
Nov 02 09:34:38.558155 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.558139    2545 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 09:34:38.561450 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:38.561412    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ba6db318717035e56c8b1001fb10e6.slice/crio-6c0719828efa2d70fa740192c6b713ae871a0172206cc80593be9364ae330fba WatchSource:0}: Error finding container 6c0719828efa2d70fa740192c6b713ae871a0172206cc80593be9364ae330fba: Status 404 returned error can't find the container with id 6c0719828efa2d70fa740192c6b713ae871a0172206cc80593be9364ae330fba
Nov 02 09:34:38.634633 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.634569    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:34:38.646674 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:38.646642    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfc6ef2_c0db_4fa7_9d91_95df203dff49.slice/crio-bdcebb3484cc8a989f0a3c91ecacda40896d2d71b700624b269e4a709259b299 WatchSource:0}: Error finding container bdcebb3484cc8a989f0a3c91ecacda40896d2d71b700624b269e4a709259b299: Status 404 returned error can't find the container with id bdcebb3484cc8a989f0a3c91ecacda40896d2d71b700624b269e4a709259b299
Nov 02 09:34:38.674323 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.674295    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:34:38.677193 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.677164    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:34:38.684709 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:38.684685    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118b62a2_be2c_4119_87d4_914839ecc25a.slice/crio-cdc069d72a1211d8c85fd25a5400db32e9412206edcdb9eb5f93661cc6fba78b WatchSource:0}: Error finding container cdc069d72a1211d8c85fd25a5400db32e9412206edcdb9eb5f93661cc6fba78b: Status 404 returned error can't find the container with id cdc069d72a1211d8c85fd25a5400db32e9412206edcdb9eb5f93661cc6fba78b
Nov 02 09:34:38.685351 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:38.685330    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77530df7_545e_42b0_ad16_ce934cb10dab.slice/crio-db15077c4cbec03878e701d7f36ac7f5f92af8065f0fbe7c233b74fc26da46b9 WatchSource:0}: Error finding container db15077c4cbec03878e701d7f36ac7f5f92af8065f0fbe7c233b74fc26da46b9: Status 404 returned error can't find the container with id db15077c4cbec03878e701d7f36ac7f5f92af8065f0fbe7c233b74fc26da46b9
Nov 02 09:34:38.702616 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.702593    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:34:38.712698 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:38.712666    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48aad7e1_623f_4eab_8a57_a50fefe7f2b1.slice/crio-0a1d871179fe9805f96aec4580350e40fb3e51ec6b03897a425ea92fc41195cf WatchSource:0}: Error finding container 0a1d871179fe9805f96aec4580350e40fb3e51ec6b03897a425ea92fc41195cf: Status 404 returned error can't find the container with id 0a1d871179fe9805f96aec4580350e40fb3e51ec6b03897a425ea92fc41195cf
Nov 02 09:34:38.736676 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.736656    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:34:38.745511 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:38.745490    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ff18f0_d0af_43a6_a5b6_ab7d2dd48deb.slice/crio-94715adf5b967fbb49956da50a9842f338aa82b3bb78e71e7340064ca9010e3c WatchSource:0}: Error finding container 94715adf5b967fbb49956da50a9842f338aa82b3bb78e71e7340064ca9010e3c: Status 404 returned error can't find the container with id 94715adf5b967fbb49956da50a9842f338aa82b3bb78e71e7340064ca9010e3c
Nov 02 09:34:38.750717 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.750699    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zhmvb"
Nov 02 09:34:38.752912 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.752897    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:34:38.761640 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:38.761617    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b69cf9_00f2_48f9_9c66_f22027a2376c.slice/crio-8aaa054d2f0b75d62b2740015291ba11af78a7a408113bc290610ad1326bf809 WatchSource:0}: Error finding container 8aaa054d2f0b75d62b2740015291ba11af78a7a408113bc290610ad1326bf809: Status 404 returned error can't find the container with id 8aaa054d2f0b75d62b2740015291ba11af78a7a408113bc290610ad1326bf809
Nov 02 09:34:38.764421 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:38.764388    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ecca6d_3d39_4f10_9129_06feb4b292fa.slice/crio-0ae26e8b8c3e1c57f571c41870d9f6a47b5673f63567f8fd52730bc733d91f77 WatchSource:0}: Error finding container 0ae26e8b8c3e1c57f571c41870d9f6a47b5673f63567f8fd52730bc733d91f77: Status 404 returned error can't find the container with id 0ae26e8b8c3e1c57f571c41870d9f6a47b5673f63567f8fd52730bc733d91f77
Nov 02 09:34:38.765044 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.765018    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:34:38.773546 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.773528    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:34:38.774184 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:38.774162    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8df50bb_8d26_42c1_b496_c9d9f56785d9.slice/crio-3ff93a6f6cbd0c3e509d86d8ab2a2ada54b2a7eb471ac8c0aadef2d8a4a58961 WatchSource:0}: Error finding container 3ff93a6f6cbd0c3e509d86d8ab2a2ada54b2a7eb471ac8c0aadef2d8a4a58961: Status 404 returned error can't find the container with id 3ff93a6f6cbd0c3e509d86d8ab2a2ada54b2a7eb471ac8c0aadef2d8a4a58961
Nov 02 09:34:38.783107 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:34:38.783088    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f5b78b_58a0_4d0d_9bf6_97c4943597fd.slice/crio-9392de63d13df86c30d47c92fc00921e42804aead1a6d0bdd53bba96c41ecd1c WatchSource:0}: Error finding container 9392de63d13df86c30d47c92fc00921e42804aead1a6d0bdd53bba96c41ecd1c: Status 404 returned error can't find the container with id 9392de63d13df86c30d47c92fc00921e42804aead1a6d0bdd53bba96c41ecd1c
Nov 02 09:34:38.884553 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.884519    2545 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:34:38.896269 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:38.896212    2545 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:34:39.031174 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.031140    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:39.031338 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.031202    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:39.031338 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:39.031295    2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:39.031461 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:39.031357    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs podName:961f499a-2ed3-4e0d-b5d2-9194d18b696a nodeName:}" failed. No retries permitted until 2026-11-02 09:34:40.031342268 +0000 UTC m=+3.143125628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs") pod "network-metrics-daemon-v9xng" (UID: "961f499a-2ed3-4e0d-b5d2-9194d18b696a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:39.031461 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:39.031302    2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:39.031461 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:39.031443    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret podName:c5b95e7b-2ce0-45a1-9859-0d2378576064 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:40.031431561 +0000 UTC m=+3.143214925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret") pod "global-pull-secret-syncer-lxn8z" (UID: "c5b95e7b-2ce0-45a1-9859-0d2378576064") : object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:39.132466 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.132424    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfq4v\" (UniqueName: \"kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v\") pod \"network-check-target-h5bzx\" (UID: \"dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e\") " pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:39.132642 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:39.132625    2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 09:34:39.132691 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:39.132652    2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 09:34:39.132691 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:39.132665    2545 projected.go:196] Error preparing data for projected volume kube-api-access-jfq4v for pod openshift-network-diagnostics/network-check-target-h5bzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:39.132757 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:39.132722    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v podName:dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e nodeName:}" failed. No retries permitted until 2026-11-02 09:34:40.132707256 +0000 UTC m=+3.244490620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfq4v" (UniqueName: "kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v") pod "network-check-target-h5bzx" (UID: "dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:39.360483 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.360023    2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 09:29:38 +0000 UTC" deadline="2028-05-02 07:11:23.476096651 +0000 UTC"
Nov 02 09:34:39.360483 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.360085    2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13125h36m44.116015458s"
Nov 02 09:34:39.448396 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.448312    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd" event={"ID":"118b62a2-be2c-4119-87d4-914839ecc25a","Type":"ContainerStarted","Data":"cdc069d72a1211d8c85fd25a5400db32e9412206edcdb9eb5f93661cc6fba78b"}
Nov 02 09:34:39.449794 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.449763    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gfgcv" event={"ID":"adfc6ef2-c0db-4fa7-9d91-95df203dff49","Type":"ContainerStarted","Data":"bdcebb3484cc8a989f0a3c91ecacda40896d2d71b700624b269e4a709259b299"}
Nov 02 09:34:39.459155 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.459123    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal" event={"ID":"33ba6db318717035e56c8b1001fb10e6","Type":"ContainerStarted","Data":"6c0719828efa2d70fa740192c6b713ae871a0172206cc80593be9364ae330fba"}
Nov 02 09:34:39.462003 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.461940    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal" event={"ID":"a2d480d50be16b70f16fe76e324fa19d","Type":"ContainerStarted","Data":"5378d6236e7ea1b301c2da5f0cd98cc06abd991f2cb580cba7d08b14abc5d3cc"}
Nov 02 09:34:39.464376 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.464345    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xzrll" event={"ID":"e8df50bb-8d26-42c1-b496-c9d9f56785d9","Type":"ContainerStarted","Data":"3ff93a6f6cbd0c3e509d86d8ab2a2ada54b2a7eb471ac8c0aadef2d8a4a58961"}
Nov 02 09:34:39.469241 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.469079    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h6h6n" event={"ID":"d9ecca6d-3d39-4f10-9129-06feb4b292fa","Type":"ContainerStarted","Data":"0ae26e8b8c3e1c57f571c41870d9f6a47b5673f63567f8fd52730bc733d91f77"}
Nov 02 09:34:39.476051 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.475969    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmqdf" event={"ID":"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb","Type":"ContainerStarted","Data":"94715adf5b967fbb49956da50a9842f338aa82b3bb78e71e7340064ca9010e3c"}
Nov 02 09:34:39.485055 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.485017    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xz8m5" event={"ID":"77530df7-545e-42b0-ad16-ce934cb10dab","Type":"ContainerStarted","Data":"db15077c4cbec03878e701d7f36ac7f5f92af8065f0fbe7c233b74fc26da46b9"}
Nov 02 09:34:39.496381 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.496352    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zz4q2" event={"ID":"92f5b78b-58a0-4d0d-9bf6-97c4943597fd","Type":"ContainerStarted","Data":"9392de63d13df86c30d47c92fc00921e42804aead1a6d0bdd53bba96c41ecd1c"}
Nov 02 09:34:39.510959 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.510919    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zhmvb" event={"ID":"09b69cf9-00f2-48f9-9c66-f22027a2376c","Type":"ContainerStarted","Data":"8aaa054d2f0b75d62b2740015291ba11af78a7a408113bc290610ad1326bf809"}
Nov 02 09:34:39.512900 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.512867    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs" event={"ID":"48aad7e1-623f-4eab-8a57-a50fefe7f2b1","Type":"ContainerStarted","Data":"0a1d871179fe9805f96aec4580350e40fb3e51ec6b03897a425ea92fc41195cf"}
Nov 02 09:34:39.824714 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.824677    2545 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:34:39.944613 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:39.944538    2545 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:34:40.041806 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:40.041759    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:40.042011 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:40.041826    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:40.042011 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:40.041958    2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:40.042123 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:40.042055    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs podName:961f499a-2ed3-4e0d-b5d2-9194d18b696a nodeName:}" failed. No retries permitted until 2026-11-02 09:34:42.042041298 +0000 UTC m=+5.153824672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs") pod "network-metrics-daemon-v9xng" (UID: "961f499a-2ed3-4e0d-b5d2-9194d18b696a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:40.042466 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:40.042323    2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:40.042466 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:40.042416    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret podName:c5b95e7b-2ce0-45a1-9859-0d2378576064 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:42.042397689 +0000 UTC m=+5.154181067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret") pod "global-pull-secret-syncer-lxn8z" (UID: "c5b95e7b-2ce0-45a1-9859-0d2378576064") : object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:40.143162 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:40.143088    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfq4v\" (UniqueName: \"kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v\") pod \"network-check-target-h5bzx\" (UID: \"dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e\") " pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:40.143297 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:40.143255    2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 09:34:40.143297 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:40.143270    2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 09:34:40.143297 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:40.143280    2545 projected.go:196] Error preparing data for projected volume kube-api-access-jfq4v for pod openshift-network-diagnostics/network-check-target-h5bzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:40.143405 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:40.143329    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v podName:dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e nodeName:}" failed. No retries permitted until 2026-11-02 09:34:42.143315568 +0000 UTC m=+5.255098928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfq4v" (UniqueName: "kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v") pod "network-check-target-h5bzx" (UID: "dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:40.360608 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:40.360561    2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 09:29:38 +0000 UTC" deadline="2028-05-24 23:08:57.981447952 +0000 UTC"
Nov 02 09:34:40.360608 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:40.360606    2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13669h34m17.620846093s"
Nov 02 09:34:40.437247 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:40.436137    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:40.437247 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:40.436276    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:34:40.437247 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:40.436842    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:40.437247 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:40.436980    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:34:40.437247 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:40.437050    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:40.437247 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:40.437130    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:34:42.060252 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:42.060212    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:42.060725 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:42.060276    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:42.060725 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:42.060415    2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:42.060725 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:42.060476    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs podName:961f499a-2ed3-4e0d-b5d2-9194d18b696a nodeName:}" failed. No retries permitted until 2026-11-02 09:34:46.060458369 +0000 UTC m=+9.172241741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs") pod "network-metrics-daemon-v9xng" (UID: "961f499a-2ed3-4e0d-b5d2-9194d18b696a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:42.060725 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:42.060539    2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:42.060725 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:42.060570    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret podName:c5b95e7b-2ce0-45a1-9859-0d2378576064 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:46.060559937 +0000 UTC m=+9.172343310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret") pod "global-pull-secret-syncer-lxn8z" (UID: "c5b95e7b-2ce0-45a1-9859-0d2378576064") : object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:42.161111 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:42.161067    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfq4v\" (UniqueName: \"kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v\") pod \"network-check-target-h5bzx\" (UID: \"dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e\") " pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:42.161287 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:42.161265    2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 09:34:42.161345 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:42.161291    2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 09:34:42.161345 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:42.161324    2545 projected.go:196] Error preparing data for projected volume kube-api-access-jfq4v for pod openshift-network-diagnostics/network-check-target-h5bzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:42.161458 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:42.161399    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v podName:dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e nodeName:}" failed. No retries permitted until 2026-11-02 09:34:46.161379904 +0000 UTC m=+9.273163262 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfq4v" (UniqueName: "kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v") pod "network-check-target-h5bzx" (UID: "dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:42.436275 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:42.435700    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:42.436275 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:42.435745    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:42.436275 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:42.435700    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:42.436275 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:42.435876    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:34:42.436275 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:42.435902    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:34:42.436275 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:42.435965    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:34:43.541870 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:43.541606    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal" event={"ID":"a2d480d50be16b70f16fe76e324fa19d","Type":"ContainerStarted","Data":"4a46a25b79cd4c2cb009f709431733c8f0c1b08945fd748b08b96c07dedfe722"}
Nov 02 09:34:43.552262 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:43.551508    2545 generic.go:356] "Generic (PLEG): container finished" podID="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" containerID="2b5b08e1e86bd1c3ce31acde64eedfe65c7e82be9b6fd013c087143a6f59225c" exitCode=0
Nov 02 09:34:43.552262 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:43.551606    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmqdf" event={"ID":"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb","Type":"ContainerDied","Data":"2b5b08e1e86bd1c3ce31acde64eedfe65c7e82be9b6fd013c087143a6f59225c"}
Nov 02 09:34:43.564364 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:43.564297    2545 generic.go:356] "Generic (PLEG): container finished" podID="33ba6db318717035e56c8b1001fb10e6" containerID="13b4563108d0a935f80344378aa8b08036712f42972aaa187e5d448253a33f87" exitCode=0
Nov 02 09:34:43.564504 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:43.564374    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal" event={"ID":"33ba6db318717035e56c8b1001fb10e6","Type":"ContainerDied","Data":"13b4563108d0a935f80344378aa8b08036712f42972aaa187e5d448253a33f87"}
Nov 02 09:34:43.587515 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:43.586791    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal" podStartSLOduration=5.586778604 podStartE2EDuration="5.586778604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:43.563054706 +0000 UTC m=+6.674838089" watchObservedRunningTime="2026-11-02 09:34:43.586778604 +0000 UTC m=+6.698561984"
Nov 02 09:34:44.437392 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:44.436505    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:44.437392 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:44.436634    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:34:44.437392 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:44.437036    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:44.437392 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:44.437137    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:34:44.437392 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:44.437206    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:44.437392 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:44.437291    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:34:44.586203 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:44.586157    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal" event={"ID":"33ba6db318717035e56c8b1001fb10e6","Type":"ContainerStarted","Data":"08eb5d5b499605bdc5cb40f03db2de845424d95bc6163569725ebec910a96774"}
Nov 02 09:34:44.589179 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:44.588649    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xzrll" event={"ID":"e8df50bb-8d26-42c1-b496-c9d9f56785d9","Type":"ContainerStarted","Data":"515cc5f869be5fa5933042ab6e365fa9b06a00594d301e0cec4e469765698cd7"}
Nov 02 09:34:44.592888 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:44.592785    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xz8m5" event={"ID":"77530df7-545e-42b0-ad16-ce934cb10dab","Type":"ContainerStarted","Data":"18c9a2093778084ffc20e0cf6fa862561b4c8c83b73b66548b85d9d67ac64ed2"}
Nov 02 09:34:44.626410 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:44.626345    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal" podStartSLOduration=6.626327809 podStartE2EDuration="6.626327809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:44.603339756 +0000 UTC m=+7.715123137" watchObservedRunningTime="2026-11-02 09:34:44.626327809 +0000 UTC m=+7.738111189"
Nov 02 09:34:44.626822 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:44.626778    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xzrll" podStartSLOduration=2.036951321 podStartE2EDuration="6.626768625s" totalImagesPullingTime="4.589817304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:44.625931567 +0000 UTC m=+7.737714949" watchObservedRunningTime="2026-11-02 09:34:44.626768625 +0000 UTC m=+7.738552009"
Nov 02 09:34:45.598809 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:45.598732    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs" event={"ID":"48aad7e1-623f-4eab-8a57-a50fefe7f2b1","Type":"ContainerStarted","Data":"87e0479cd941835dc8e1216908b8e1e9640c7260aeba56f1ea2ea124eb4ad17c"}
Nov 02 09:34:46.090762 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:46.090719    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:46.090927 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:46.090796    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:46.090985 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:46.090938    2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:46.091034 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:46.091003    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs podName:961f499a-2ed3-4e0d-b5d2-9194d18b696a nodeName:}" failed. No retries permitted until 2026-11-02 09:34:54.090983986 +0000 UTC m=+17.202767414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs") pod "network-metrics-daemon-v9xng" (UID: "961f499a-2ed3-4e0d-b5d2-9194d18b696a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:46.091460 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:46.091435    2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:46.091582 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:46.091498    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret podName:c5b95e7b-2ce0-45a1-9859-0d2378576064 nodeName:}" failed. No retries permitted until 2026-11-02 09:34:54.09148226 +0000 UTC m=+17.203265624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret") pod "global-pull-secret-syncer-lxn8z" (UID: "c5b95e7b-2ce0-45a1-9859-0d2378576064") : object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:46.191955 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:46.191877    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfq4v\" (UniqueName: \"kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v\") pod \"network-check-target-h5bzx\" (UID: \"dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e\") " pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:46.192114 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:46.192057    2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 09:34:46.192114 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:46.192093    2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 09:34:46.192114 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:46.192107    2545 projected.go:196] Error preparing data for projected volume kube-api-access-jfq4v for pod openshift-network-diagnostics/network-check-target-h5bzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:46.192300 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:46.192166    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v podName:dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e nodeName:}" failed. No retries permitted until 2026-11-02 09:34:54.192148994 +0000 UTC m=+17.303932365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfq4v" (UniqueName: "kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v") pod "network-check-target-h5bzx" (UID: "dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:46.435838 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:46.435804    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:46.436023 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:46.435904    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:34:46.436023 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:46.435967    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:46.436023 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:46.436017    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:34:46.436242 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:46.436051    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:46.436242 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:46.436094    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:34:46.844980 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:46.844954    2545 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 09:34:47.367375 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:47.363781    2545 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T09:34:46.844976888Z","UUID":"b7ae798b-bc70-4caf-baf8-b97a85c93367","Handler":null,"Name":"","Endpoint":""}
Nov 02 09:34:47.367375 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:47.365804    2545 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 09:34:47.367375 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:47.365845    2545 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 09:34:47.604310 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:47.604211    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zz4q2" event={"ID":"92f5b78b-58a0-4d0d-9bf6-97c4943597fd","Type":"ContainerStarted","Data":"ea0694ccc10ec1cc9b78e64e2779755ebbbd920adb9e8f5e1e63f0bb22da9159"}
Nov 02 09:34:47.606618 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:47.606586    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zhmvb" event={"ID":"09b69cf9-00f2-48f9-9c66-f22027a2376c","Type":"ContainerStarted","Data":"a9633984179408c8001a0a409cf218602a6601d9e1a185ef162aab97b84e755d"}
Nov 02 09:34:47.609510 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:47.609481    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs" event={"ID":"48aad7e1-623f-4eab-8a57-a50fefe7f2b1","Type":"ContainerStarted","Data":"3beb8deccf11d66dec6174a3eb093e5c4fcf95c46aca64e09f3a3024efa87966"}
Nov 02 09:34:47.625531 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:47.625054    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zz4q2" podStartSLOduration=1.301832061 podStartE2EDuration="9.625037886s" totalImagesPullingTime="8.323205825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:47.624493661 +0000 UTC m=+10.736277042" watchObservedRunningTime="2026-11-02 09:34:47.625037886 +0000 UTC m=+10.736821268"
Nov 02 09:34:47.625531 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:47.625153    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xz8m5" podStartSLOduration=5.155724368 podStartE2EDuration="10.625147081s" totalImagesPullingTime="5.469422713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:44.642422648 +0000 UTC m=+7.754206041" watchObservedRunningTime="2026-11-02 09:34:47.625147081 +0000 UTC m=+10.736930461"
Nov 02 09:34:47.645958 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:47.645906    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zhmvb" podStartSLOduration=2.532775596 podStartE2EDuration="10.645888628s" totalImagesPullingTime="8.113113032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:47.645053876 +0000 UTC m=+10.756837255" watchObservedRunningTime="2026-11-02 09:34:47.645888628 +0000 UTC m=+10.757672010"
Nov 02 09:34:48.435714 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:48.435678    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:48.435714 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:48.435698    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:48.436130 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:48.435877    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:34:48.436130 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:48.435934    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:48.436130 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:48.436050    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:34:48.436252 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:48.436143    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:34:48.613718 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:48.613682    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h6h6n" event={"ID":"d9ecca6d-3d39-4f10-9129-06feb4b292fa","Type":"ContainerStarted","Data":"582409d442c0417e41ca0962d5af92bf7ba2b608acab6891f732379012a1f9ee"}
Nov 02 09:34:48.615948 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:48.615921    2545 generic.go:356] "Generic (PLEG): container finished" podID="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" containerID="4d08032f9df97d5ae625422e9277c2cb4c734af3d38dbc6cd7c758ebe5917915" exitCode=0
Nov 02 09:34:48.616086 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:48.615999    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmqdf" event={"ID":"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb","Type":"ContainerDied","Data":"4d08032f9df97d5ae625422e9277c2cb4c734af3d38dbc6cd7c758ebe5917915"}
Nov 02 09:34:48.619753 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:48.619727    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs" event={"ID":"48aad7e1-623f-4eab-8a57-a50fefe7f2b1","Type":"ContainerStarted","Data":"dd106919a3787ee598ed3aef2441ecdd75256457beb6057e1b78c3253a794427"}
Nov 02 09:34:48.631859 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:48.631815    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h6h6n" podStartSLOduration=3.039394191 podStartE2EDuration="11.631799732s" totalImagesPullingTime="8.592405541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:48.631393113 +0000 UTC m=+11.743176495" watchObservedRunningTime="2026-11-02 09:34:48.631799732 +0000 UTC m=+11.743583114"
Nov 02 09:34:48.681847 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:48.681790    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs" podStartSLOduration=2.840156921 podStartE2EDuration="11.681773276s" totalImagesPullingTime="8.841616355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:48.681373107 +0000 UTC m=+11.793156490" watchObservedRunningTime="2026-11-02 09:34:48.681773276 +0000 UTC m=+11.793556655"
Nov 02 09:34:49.408792 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:49.408507    2545 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:34:49.409288 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:49.409267    2545 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:34:49.622742 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:49.622715    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gfgcv" event={"ID":"adfc6ef2-c0db-4fa7-9d91-95df203dff49","Type":"ContainerStarted","Data":"a03b8bafb430729fd723b28efb197ea80a603dd30a371189c643a28906eb7e88"}
Nov 02 09:34:49.645209 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:49.645170    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gfgcv" podStartSLOduration=2.269302846 podStartE2EDuration="12.645156168s" totalImagesPullingTime="10.375853322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:49.645058684 +0000 UTC m=+12.756842064" watchObservedRunningTime="2026-11-02 09:34:49.645156168 +0000 UTC m=+12.756939549"
Nov 02 09:34:50.435553 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:50.435516    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:50.435553 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:50.435545    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:50.435553 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:50.435556    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:50.435786 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:50.435625    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:34:50.435786 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:50.435670    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:34:50.435786 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:50.435728    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:34:50.628929 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:50.628886    2545 generic.go:356] "Generic (PLEG): container finished" podID="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" containerID="08d5d33f2a07c85f1d08fe4fdf61a12a3d35b547784bd7cb0f75ac8e2b8cd765" exitCode=0
Nov 02 09:34:50.629570 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:50.629547    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmqdf" event={"ID":"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb","Type":"ContainerDied","Data":"08d5d33f2a07c85f1d08fe4fdf61a12a3d35b547784bd7cb0f75ac8e2b8cd765"}
Nov 02 09:34:52.436013 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:52.435978    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:52.436767 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:52.436103    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:52.436767 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:52.436109    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:34:52.436767 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:52.436240    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:34:52.436767 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:52.436292    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:52.436767 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:52.436380    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:34:52.636183 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:52.636146    2545 generic.go:356] "Generic (PLEG): container finished" podID="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" containerID="02dbae894a922a02f9defca24ee501c916752da1d483975d009b58a68a43ddb4" exitCode=0
Nov 02 09:34:52.636330 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:52.636205    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmqdf" event={"ID":"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb","Type":"ContainerDied","Data":"02dbae894a922a02f9defca24ee501c916752da1d483975d009b58a68a43ddb4"}
Nov 02 09:34:54.152618 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:54.152390    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:54.153098 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:54.152646    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:54.153098 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:54.152560    2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:54.153098 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:54.152780    2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:54.153098 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:54.152782    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret podName:c5b95e7b-2ce0-45a1-9859-0d2378576064 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:10.152745237 +0000 UTC m=+33.264528613 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret") pod "global-pull-secret-syncer-lxn8z" (UID: "c5b95e7b-2ce0-45a1-9859-0d2378576064") : object "kube-system"/"original-pull-secret" not registered
Nov 02 09:34:54.153098 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:54.152849    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs podName:961f499a-2ed3-4e0d-b5d2-9194d18b696a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:10.15283339 +0000 UTC m=+33.264616748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs") pod "network-metrics-daemon-v9xng" (UID: "961f499a-2ed3-4e0d-b5d2-9194d18b696a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 09:34:54.253780 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:54.253741    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfq4v\" (UniqueName: \"kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v\") pod \"network-check-target-h5bzx\" (UID: \"dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e\") " pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:54.253964 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:54.253889    2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 09:34:54.253964 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:54.253910    2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 09:34:54.253964 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:54.253923    2545 projected.go:196] Error preparing data for projected volume kube-api-access-jfq4v for pod openshift-network-diagnostics/network-check-target-h5bzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:54.254123 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:54.253983    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v podName:dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e nodeName:}" failed. No retries permitted until 2026-11-02 09:35:10.253965896 +0000 UTC m=+33.365749273 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfq4v" (UniqueName: "kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v") pod "network-check-target-h5bzx" (UID: "dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 09:34:54.436871 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:54.436345    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:54.436871 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:54.436373    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:54.436871 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:54.436525    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:54.436871 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:54.436485    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:34:54.436871 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:54.436603    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:34:54.436871 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:54.436685    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:34:55.645194 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:55.645170    2545 generic.go:356] "Generic (PLEG): container finished" podID="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" containerID="e099aba2156a917faa484c556c599578fec2ab59ae9f47d247dbe23ecbf94676" exitCode=0
Nov 02 09:34:55.645558 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:55.645217    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmqdf" event={"ID":"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb","Type":"ContainerDied","Data":"e099aba2156a917faa484c556c599578fec2ab59ae9f47d247dbe23ecbf94676"}
Nov 02 09:34:56.436473 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:56.436439    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:56.436646 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:56.436448    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:56.436646 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:56.436543    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:34:56.436646 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:56.436562    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:56.436646 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:56.436640    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:34:56.436768 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:56.436703    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:34:56.650453 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:56.650423    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd" event={"ID":"118b62a2-be2c-4119-87d4-914839ecc25a","Type":"ContainerStarted","Data":"e415f2d10ee34b043d850f5d9d5fe844501799c64af8afb5734312c1171ecf7e"}
Nov 02 09:34:56.650831 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:56.650490    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd" event={"ID":"118b62a2-be2c-4119-87d4-914839ecc25a","Type":"ContainerStarted","Data":"c527275d88a05109a10a69af5cc42f91079c4b5a4a113568274eebfed3c10768"}
Nov 02 09:34:56.650831 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:56.650540    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd" event={"ID":"118b62a2-be2c-4119-87d4-914839ecc25a","Type":"ContainerStarted","Data":"67345efa7b727d03454fa2fe92375e81f4fb6325a3b42ff1888b12687bdab84d"}
Nov 02 09:34:56.650831 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:56.650571    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd" event={"ID":"118b62a2-be2c-4119-87d4-914839ecc25a","Type":"ContainerStarted","Data":"a15691f8385363894f3edf923bc5876ca30321d4ae863721e5c50036e415d4ce"}
Nov 02 09:34:56.650831 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:56.650597    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd" event={"ID":"118b62a2-be2c-4119-87d4-914839ecc25a","Type":"ContainerStarted","Data":"d839776bc1ff3873162ed9cf51935aa25209ce474fb43b6489c3e9af10b1cfde"}
Nov 02 09:34:56.650831 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:56.650638    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd" event={"ID":"118b62a2-be2c-4119-87d4-914839ecc25a","Type":"ContainerStarted","Data":"cde6d0bdad2a99b1596442dcc9820850a8825aca50b78eb718ff5179d4a71e65"}
Nov 02 09:34:56.653220 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:56.653196    2545 generic.go:356] "Generic (PLEG): container finished" podID="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" containerID="4680a323d324a457e2e27fc4e876588ba1515af257b4b410001ce85e951230d4" exitCode=0
Nov 02 09:34:56.653328 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:56.653230    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmqdf" event={"ID":"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb","Type":"ContainerDied","Data":"4680a323d324a457e2e27fc4e876588ba1515af257b4b410001ce85e951230d4"}
Nov 02 09:34:57.659028 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:57.658838    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmqdf" event={"ID":"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb","Type":"ContainerStarted","Data":"d0995fc556aef9cb24e12adc3ea14f8c0fbf3f36ff2aeb97a1a7df4154c83fc6"}
Nov 02 09:34:57.683422 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:57.683381    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kmqdf" podStartSLOduration=9.148943113 podStartE2EDuration="20.68336944s" totalImagesPullingTime="11.534426327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:34:57.682874745 +0000 UTC m=+20.794658125" watchObservedRunningTime="2026-11-02 09:34:57.68336944 +0000 UTC m=+20.795152831"
Nov 02 09:34:58.436521 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:58.436434    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:34:58.436521 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:58.436461    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:34:58.436697 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:58.436564    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:34:58.436697 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:58.436608    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:34:58.436697 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:58.436686    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:34:58.436800 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:34:58.436760    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:34:58.663779 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:34:58.663749    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd" event={"ID":"118b62a2-be2c-4119-87d4-914839ecc25a","Type":"ContainerStarted","Data":"de0e156db93436919ed528d1cb40fa1730cb189b083a1dc9ddc21c5b742d08f2"}
Nov 02 09:35:00.435982 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:00.435900    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:00.436721 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:00.436005    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:00.436721 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:00.436096    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:35:00.436721 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:00.436012    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:35:00.436721 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:00.436011    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:35:00.436721 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:00.436180    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:35:00.672699 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:00.672665    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd" event={"ID":"118b62a2-be2c-4119-87d4-914839ecc25a","Type":"ContainerStarted","Data":"0637813cd94f475c591254417e0556e1e5004b0996258d54c8fb871a8c662dee"}
Nov 02 09:35:00.672910 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:00.672893    2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:00.687756 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:00.687694    2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:00.704371 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:00.704311    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd" podStartSLOduration=6.753480778 podStartE2EDuration="23.704300279s" totalImagesPullingTime="16.950819501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 09:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:00.702507415 +0000 UTC m=+23.814290796" watchObservedRunningTime="2026-11-02 09:35:00.704300279 +0000 UTC m=+23.816083660"
Nov 02 09:35:01.675299 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:01.675271    2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:01.675906 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:01.675366    2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:01.689381 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:01.689342    2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:01.818793 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:01.818207    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lxn8z"]
Nov 02 09:35:01.818793 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:01.818341    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:01.818793 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:01.818473    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:35:01.819016 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:01.818937    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h5bzx"]
Nov 02 09:35:01.819048 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:01.819022    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:35:01.819106 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:01.819081    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:35:01.820051 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:01.820031    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v9xng"]
Nov 02 09:35:01.820141 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:01.820130    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:01.820246 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:01.820230    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:35:03.436611 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:03.436579    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:03.437100 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:03.436615    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:03.437100 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:03.436668    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:35:03.437100 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:03.436704    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:35:03.437100 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:03.436779    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:35:03.437100 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:03.436834    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:35:05.435834 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:05.435799    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:05.436226 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:05.435799    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:35:05.436226 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:05.435930    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:35:05.436226 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:05.435799    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:05.436226 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:05.435992    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:35:05.436226 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:05.436079    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:35:06.259606 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:06.259571    2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:35:06.259742 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:06.259713    2545 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 09:35:06.260260 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:06.260243    2545 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:35:07.436965 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:07.436928    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:07.437440 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:07.437040    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:07.437440 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:07.437062    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lxn8z" podUID="c5b95e7b-2ce0-45a1-9859-0d2378576064"
Nov 02 09:35:07.437440 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:07.437103    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v9xng" podUID="961f499a-2ed3-4e0d-b5d2-9194d18b696a"
Nov 02 09:35:07.437440 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:07.437116    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:35:07.437440 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:07.437179    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5bzx" podUID="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e"
Nov 02 09:35:08.712978 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.712951    2545 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeReady"
Nov 02 09:35:08.713449 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.713063    2545 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 09:35:08.749813 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.749785    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pprd9"]
Nov 02 09:35:08.820973 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.820945    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cf7t5"]
Nov 02 09:35:08.821115 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.821071    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pprd9"
Nov 02 09:35:08.827028 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.827000    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 09:35:08.827152 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.827067    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 09:35:08.839810 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.839787    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xtlc6"]
Nov 02 09:35:08.839922 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.839907    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cf7t5"
Nov 02 09:35:08.852427 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.852410    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bc8fbd47b-96z8h"]
Nov 02 09:35:08.852537 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.852522    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xtlc6"
Nov 02 09:35:08.866488 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.866471    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-pdw8n"]
Nov 02 09:35:08.866577 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.866562    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:08.870368 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.870339    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 09:35:08.870368 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.870367    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 09:35:08.870530 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.870515    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 09:35:08.870577 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.870534    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cmxq6\""
Nov 02 09:35:08.880618 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.880601    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-68d88c448f-z5k5p"]
Nov 02 09:35:08.880714 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.880704    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:08.884110 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.884092    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 09:35:08.884233 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.884220    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fzpc9\""
Nov 02 09:35:08.891855 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.891839    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"]
Nov 02 09:35:08.891963 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.891943    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:08.911994 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.911975    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"]
Nov 02 09:35:08.912099 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.912085    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:08.928130 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.928109    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79578858d5-6w82j"]
Nov 02 09:35:08.929456 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.929429    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:08.930182 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.930149    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 09:35:08.930278 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.930202    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 09:35:08.930278 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.930261    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 09:35:08.930369 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.930281    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 09:35:08.930369 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.930283    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 09:35:08.930369 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.930284    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 09:35:08.930691 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.930675    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 09:35:08.930824 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.930805    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 09:35:08.931154 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.931136    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 09:35:08.933614 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.933592    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:08.933699 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.933618    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 09:35:08.933985 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.933962    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 09:35:08.934550 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.934536    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 09:35:08.934917 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.934899    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 09:35:08.949101 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.949084    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-n99gd"]
Nov 02 09:35:08.949190 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.949177    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:08.965817 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.965776    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh7fg\" (UniqueName: \"kubernetes.io/projected/fce2663d-12a8-427f-b8ec-0d2238a06470-kube-api-access-qh7fg\") pod \"kas-connection-checker-685895c7c9-xtlc6\" (UID: \"fce2663d-12a8-427f-b8ec-0d2238a06470\") " pod="kube-system/kas-connection-checker-685895c7c9-xtlc6"
Nov 02 09:35:08.965817 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.965803    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-trusted-ca\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:08.965926 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.965824    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85741a00-b702-46cb-a90c-e000bf9976f0-snapshots\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:08.965926 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.965841    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:08.965926 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.965880    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvdcd\" (UniqueName: \"kubernetes.io/projected/95d7a9df-37e3-492b-9540-3df603931723-kube-api-access-qvdcd\") pod \"kas-connection-checker-685895c7c9-pprd9\" (UID: \"95d7a9df-37e3-492b-9540-3df603931723\") " pod="kube-system/kas-connection-checker-685895c7c9-pprd9"
Nov 02 09:35:08.965926 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.965910    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-certificates\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:08.966042 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.966006    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2daf8a59-8027-42b4-8414-55a0883bc04f-ca-trust-extracted\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:08.966042 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.966028    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-installation-pull-secrets\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:08.966107 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.966094    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5ql9\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-kube-api-access-v5ql9\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:08.966137 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.966124    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psmww\" (UniqueName: \"kubernetes.io/projected/78cde7e0-a2f3-4f33-ba88-6c9fedc723eb-kube-api-access-psmww\") pod \"kas-connection-checker-685895c7c9-cf7t5\" (UID: \"78cde7e0-a2f3-4f33-ba88-6c9fedc723eb\") " pod="kube-system/kas-connection-checker-685895c7c9-cf7t5"
Nov 02 09:35:08.966168 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.966138    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85741a00-b702-46cb-a90c-e000bf9976f0-service-ca-bundle\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:08.966168 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.966161    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b25xs\" (UniqueName: \"kubernetes.io/projected/85741a00-b702-46cb-a90c-e000bf9976f0-kube-api-access-b25xs\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:08.966240 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.966227    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85741a00-b702-46cb-a90c-e000bf9976f0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:08.966269 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.966247    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-bound-sa-token\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:08.966269 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.966264    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85741a00-b702-46cb-a90c-e000bf9976f0-tmp\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:08.966328 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.966278    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-image-registry-private-configuration\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:08.966374 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.966347    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85741a00-b702-46cb-a90c-e000bf9976f0-serving-cert\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:08.970693 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.970670    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 09:35:08.970781 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.970718    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 09:35:08.970781 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.970725    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 09:35:08.970781 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.970758    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 09:35:08.970930 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.970789    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 09:35:08.970930 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.970810    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pcvs5\""
Nov 02 09:35:08.970930 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.970811    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-68wrf\""
Nov 02 09:35:08.970930 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.970842    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"]
Nov 02 09:35:08.971087 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.970970    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:08.971488 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.971472    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bhljs\""
Nov 02 09:35:08.974418 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.974401    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 09:35:08.974576 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.974563    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hrc8h\""
Nov 02 09:35:08.990759 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.990742    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sb5ns"]
Nov 02 09:35:08.990891 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:08.990877    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:09.014693 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.014669    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"]
Nov 02 09:35:09.014799 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.014777    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sb5ns"
Nov 02 09:35:09.027807 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.027788    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 09:35:09.027912 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.027853    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 09:35:09.027912 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.027858    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:09.027912 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.027870    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 09:35:09.027912 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.027911    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 09:35:09.028058 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.028046    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:09.028058 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.028053    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 09:35:09.028141 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.028055    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:09.030130 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.030037    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 09:35:09.037805 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.037787    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"]
Nov 02 09:35:09.037937 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.037922    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:09.048639 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.048623    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mvxdw"]
Nov 02 09:35:09.048936 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.048914    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:09.062976 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.062960    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pprd9"]
Nov 02 09:35:09.063053 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.062988    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cf7t5"]
Nov 02 09:35:09.063053 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.063007    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-pdw8n"]
Nov 02 09:35:09.063053 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.063028    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"]
Nov 02 09:35:09.063053 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.063046    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"]
Nov 02 09:35:09.063170 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.063066    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mvxdw"]
Nov 02 09:35:09.063170 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.063084    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bc8fbd47b-96z8h"]
Nov 02 09:35:09.063170 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.063085    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvxdw"
Nov 02 09:35:09.063170 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.063104    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sb5ns"]
Nov 02 09:35:09.063170 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.063123    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-n99gd"]
Nov 02 09:35:09.063170 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.063142    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79578858d5-6w82j"]
Nov 02 09:35:09.063170 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.063164    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xkrcs"]
Nov 02 09:35:09.067427 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067409    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-trusted-ca\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.067488 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067433    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-installation-pull-secrets\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.067488 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067454    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvdcd\" (UniqueName: \"kubernetes.io/projected/95d7a9df-37e3-492b-9540-3df603931723-kube-api-access-qvdcd\") pod \"kas-connection-checker-685895c7c9-pprd9\" (UID: \"95d7a9df-37e3-492b-9540-3df603931723\") " pod="kube-system/kas-connection-checker-685895c7c9-pprd9"
Nov 02 09:35:09.067488 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067469    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-image-registry-private-configuration\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.067641 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067488    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:09.067641 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067589    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ql9\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-kube-api-access-v5ql9\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.067641 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067630    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.067791 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067736    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-default-certificate\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.067791 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067761    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psmww\" (UniqueName: \"kubernetes.io/projected/78cde7e0-a2f3-4f33-ba88-6c9fedc723eb-kube-api-access-psmww\") pod \"kas-connection-checker-685895c7c9-cf7t5\" (UID: \"78cde7e0-a2f3-4f33-ba88-6c9fedc723eb\") " pod="kube-system/kas-connection-checker-685895c7c9-cf7t5"
Nov 02 09:35:09.067791 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067780    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85741a00-b702-46cb-a90c-e000bf9976f0-service-ca-bundle\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:09.067900 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067798    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-image-registry-private-configuration\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.067900 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067851    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-bound-sa-token\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.067900 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067869    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d28887e-4aa9-4209-a51a-fec831136660-trusted-ca\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:09.067900 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067885    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8ms9\" (UniqueName: \"kubernetes.io/projected/2d28887e-4aa9-4209-a51a-fec831136660-kube-api-access-n8ms9\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:09.068082 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067906    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2608084-c9df-4998-a521-fb2239b3c349-config\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:09.068082 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067923    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d28887e-4aa9-4209-a51a-fec831136660-serving-cert\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:09.068082 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067940    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-certificates\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.068082 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067958    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7fg\" (UniqueName: \"kubernetes.io/projected/fce2663d-12a8-427f-b8ec-0d2238a06470-kube-api-access-qh7fg\") pod \"kas-connection-checker-685895c7c9-xtlc6\" (UID: \"fce2663d-12a8-427f-b8ec-0d2238a06470\") " pod="kube-system/kas-connection-checker-685895c7c9-xtlc6"
Nov 02 09:35:09.068082 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067974    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4zq4\" (UniqueName: \"kubernetes.io/projected/80746de3-f9f3-4efe-a963-9b37f290fcb7-kube-api-access-q4zq4\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.068082 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.067993    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85741a00-b702-46cb-a90c-e000bf9976f0-snapshots\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:09.068082 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.068007    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2608084-c9df-4998-a521-fb2239b3c349-serving-cert\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:09.068550 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.068485    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85741a00-b702-46cb-a90c-e000bf9976f0-service-ca-bundle\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:09.068550 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.068540    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.068659 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.068581    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-certificates\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.068659 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.068641    2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:09.068659 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.068653    2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8fbd47b-96z8h: secret "image-registry-tls" not found
Nov 02 09:35:09.068813 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.068695    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls podName:2daf8a59-8027-42b4-8414-55a0883bc04f nodeName:}" failed. No retries permitted until 2026-11-02 09:35:09.568679355 +0000 UTC m=+32.680462713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls") pod "image-registry-bc8fbd47b-96z8h" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f") : secret "image-registry-tls" not found
Nov 02 09:35:09.068813 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.068731    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.068813 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.068751    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3603dee5-a95f-47e4-9d60-381e82b8afa4-ca-trust-extracted\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.068813 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.068802    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85741a00-b702-46cb-a90c-e000bf9976f0-snapshots\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:09.069026 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.068855    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2daf8a59-8027-42b4-8414-55a0883bc04f-ca-trust-extracted\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.069026 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.068891    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-installation-pull-secrets\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.069026 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.068959    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.069026 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069018    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzrxx\" (UniqueName: \"kubernetes.io/projected/a2608084-c9df-4998-a521-fb2239b3c349-kube-api-access-zzrxx\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:09.069224 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069083    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3c01c082-8479-460d-a80d-5e590ab2fcbe-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:09.069224 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069129    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b25xs\" (UniqueName: \"kubernetes.io/projected/85741a00-b702-46cb-a90c-e000bf9976f0-kube-api-access-b25xs\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:09.069224 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069193    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2daf8a59-8027-42b4-8414-55a0883bc04f-ca-trust-extracted\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.069224 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069215    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85741a00-b702-46cb-a90c-e000bf9976f0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:09.069433 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069283    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-certificates\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.069433 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069374    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85741a00-b702-46cb-a90c-e000bf9976f0-tmp\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:09.069433 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069423    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85741a00-b702-46cb-a90c-e000bf9976f0-serving-cert\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:09.069579 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069458    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-bound-sa-token\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.069579 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069483    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhsmx\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-kube-api-access-xhsmx\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.069579 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069507    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b2xg\" (UniqueName: \"kubernetes.io/projected/3c01c082-8479-460d-a80d-5e590ab2fcbe-kube-api-access-7b2xg\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:09.069579 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069535    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-trusted-ca\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.069579 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069562    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d28887e-4aa9-4209-a51a-fec831136660-config\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:09.070260 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069584    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-stats-auth\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.070260 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.069627    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85741a00-b702-46cb-a90c-e000bf9976f0-tmp\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:09.070260 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.070166    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85741a00-b702-46cb-a90c-e000bf9976f0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:09.070814 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.070749    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-trusted-ca\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.071237 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.071193    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 09:35:09.071534 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.071437    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 09:35:09.071534 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.071462    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 09:35:09.071534 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.071488    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-w8zs4\""
Nov 02 09:35:09.071534 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.071502    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-66z6l\""
Nov 02 09:35:09.071534 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.071510    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bs7z2\""
Nov 02 09:35:09.071795 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.071568    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-q24jz\""
Nov 02 09:35:09.071795 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.071575    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jgpb4\""
Nov 02 09:35:09.072095 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.072074    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-installation-pull-secrets\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.072166 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.072075    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-image-registry-private-configuration\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.073533 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.073513    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85741a00-b702-46cb-a90c-e000bf9976f0-serving-cert\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:09.079810 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.079793    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvdcd\" (UniqueName: \"kubernetes.io/projected/95d7a9df-37e3-492b-9540-3df603931723-kube-api-access-qvdcd\") pod \"kas-connection-checker-685895c7c9-pprd9\" (UID: \"95d7a9df-37e3-492b-9540-3df603931723\") " pod="kube-system/kas-connection-checker-685895c7c9-pprd9"
Nov 02 09:35:09.081123 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.081096    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ql9\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-kube-api-access-v5ql9\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.081346 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.081330    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psmww\" (UniqueName: \"kubernetes.io/projected/78cde7e0-a2f3-4f33-ba88-6c9fedc723eb-kube-api-access-psmww\") pod \"kas-connection-checker-685895c7c9-cf7t5\" (UID: \"78cde7e0-a2f3-4f33-ba88-6c9fedc723eb\") " pod="kube-system/kas-connection-checker-685895c7c9-cf7t5"
Nov 02 09:35:09.081454 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.081439    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7fg\" (UniqueName: \"kubernetes.io/projected/fce2663d-12a8-427f-b8ec-0d2238a06470-kube-api-access-qh7fg\") pod \"kas-connection-checker-685895c7c9-xtlc6\" (UID: \"fce2663d-12a8-427f-b8ec-0d2238a06470\") " pod="kube-system/kas-connection-checker-685895c7c9-xtlc6"
Nov 02 09:35:09.081562 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.081545    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-bound-sa-token\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.081927 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.081913    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b25xs\" (UniqueName: \"kubernetes.io/projected/85741a00-b702-46cb-a90c-e000bf9976f0-kube-api-access-b25xs\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:09.090049 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.090023    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zgjvs"]
Nov 02 09:35:09.090156 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.090137    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xkrcs"
Nov 02 09:35:09.110308 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.110292    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xtlc6"]
Nov 02 09:35:09.110406 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.110319    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"]
Nov 02 09:35:09.110406 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.110333    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"]
Nov 02 09:35:09.110406 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.110348    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68d88c448f-z5k5p"]
Nov 02 09:35:09.110406 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.110378    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"]
Nov 02 09:35:09.110406 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.110396    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xkrcs"]
Nov 02 09:35:09.110406 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.110408    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zgjvs"]
Nov 02 09:35:09.110602 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.110437    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:09.131969 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.131946    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 09:35:09.132060 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.132045    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 09:35:09.132117 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.132048    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:09.132117 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.132074    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 09:35:09.132117 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.132081    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 09:35:09.132117 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.132085    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 09:35:09.132265 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.132141    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 09:35:09.132265 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.132142    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 09:35:09.132265 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.132194    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 09:35:09.147363 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.147338    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pprd9"
Nov 02 09:35:09.160578 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.160556    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cf7t5"
Nov 02 09:35:09.169937 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.169916    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4zq4\" (UniqueName: \"kubernetes.io/projected/80746de3-f9f3-4efe-a963-9b37f290fcb7-kube-api-access-q4zq4\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.170015 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.169946    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2608084-c9df-4998-a521-fb2239b3c349-serving-cert\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:09.170015 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.169969    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8htqg\" (UniqueName: \"kubernetes.io/projected/a05b1aec-cdf3-424b-8dfe-d4a6f06d6536-kube-api-access-8htqg\") pod \"network-check-source-6f78d59f99-mvxdw\" (UID: \"a05b1aec-cdf3-424b-8dfe-d4a6f06d6536\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvxdw"
Nov 02 09:35:09.170107 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170084    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86xb\" (UniqueName: \"kubernetes.io/projected/dbd4d998-a5be-4a0a-9113-8f3782df753a-kube-api-access-b86xb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:09.170297 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170122    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.170297 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170148    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3603dee5-a95f-47e4-9d60-381e82b8afa4-ca-trust-extracted\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.170297 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170191    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.170297 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.170255    2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:09.170297 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.170272    2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79578858d5-6w82j: secret "image-registry-tls" not found
Nov 02 09:35:09.170297 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.170274    2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 09:35:09.170297 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170273    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbd4d998-a5be-4a0a-9113-8f3782df753a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:09.170645 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.170321    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls podName:3603dee5-a95f-47e4-9d60-381e82b8afa4 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:09.670302191 +0000 UTC m=+32.782085559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls") pod "image-registry-79578858d5-6w82j" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4") : secret "image-registry-tls" not found
Nov 02 09:35:09.170645 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170341    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzrxx\" (UniqueName: \"kubernetes.io/projected/a2608084-c9df-4998-a521-fb2239b3c349-kube-api-access-zzrxx\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:09.170645 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.170397    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:09.67037803 +0000 UTC m=+32.782161394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : secret "router-metrics-certs-default" not found
Nov 02 09:35:09.170645 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170439    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsbgj\" (UniqueName: \"kubernetes.io/projected/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-kube-api-access-zsbgj\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:09.170645 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170484    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3c01c082-8479-460d-a80d-5e590ab2fcbe-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:09.170645 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170552    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmw7c\" (UniqueName: \"kubernetes.io/projected/b9ed0d41-1122-4ec2-a9af-e84cc71605df-kube-api-access-pmw7c\") pod \"volume-data-source-validator-58c9f5f46d-sb5ns\" (UID: \"b9ed0d41-1122-4ec2-a9af-e84cc71605df\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sb5ns"
Nov 02 09:35:09.170645 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170576    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr6kp\" (UniqueName: \"kubernetes.io/projected/26adcc96-ef0b-416f-8b15-e01be59fb5ae-kube-api-access-zr6kp\") pod \"ingress-canary-xkrcs\" (UID: \"26adcc96-ef0b-416f-8b15-e01be59fb5ae\") " pod="openshift-ingress-canary/ingress-canary-xkrcs"
Nov 02 09:35:09.170645 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170602    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:09.170645 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170641    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e5393080-e20a-47fc-8dc7-125f42457736-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:09.171054 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170713    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-bound-sa-token\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.171054 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170746    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhsmx\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-kube-api-access-xhsmx\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.171054 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170863    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3603dee5-a95f-47e4-9d60-381e82b8afa4-ca-trust-extracted\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.171054 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.170866    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b2xg\" (UniqueName: \"kubernetes.io/projected/3c01c082-8479-460d-a80d-5e590ab2fcbe-kube-api-access-7b2xg\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:09.171054 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171011    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d28887e-4aa9-4209-a51a-fec831136660-config\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:09.171054 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171041    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-stats-auth\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.171326 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171066    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-trusted-ca\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.171326 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171092    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-installation-pull-secrets\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.171326 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171120    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-image-registry-private-configuration\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.171326 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171146    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26adcc96-ef0b-416f-8b15-e01be59fb5ae-cert\") pod \"ingress-canary-xkrcs\" (UID: \"26adcc96-ef0b-416f-8b15-e01be59fb5ae\") " pod="openshift-ingress-canary/ingress-canary-xkrcs"
Nov 02 09:35:09.171326 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171176    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:09.171326 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171208    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.171326 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171233    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-default-certificate\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.171326 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171260    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbd4d998-a5be-4a0a-9113-8f3782df753a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:09.171326 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171294    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:09.171760 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171386    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d28887e-4aa9-4209-a51a-fec831136660-trusted-ca\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:09.171760 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171390    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3c01c082-8479-460d-a80d-5e590ab2fcbe-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:09.171760 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171425    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ms9\" (UniqueName: \"kubernetes.io/projected/2d28887e-4aa9-4209-a51a-fec831136660-kube-api-access-n8ms9\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:09.171760 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171452    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2608084-c9df-4998-a521-fb2239b3c349-config\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:09.171760 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171478    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:09.171760 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171503    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d28887e-4aa9-4209-a51a-fec831136660-serving-cert\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:09.171760 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171527    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-certificates\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.172074 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.171776    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d28887e-4aa9-4209-a51a-fec831136660-config\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:09.172074 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.172002    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zwj6d\""
Nov 02 09:35:09.172074 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.172037    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-certificates\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.172074 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.172072    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 09:35:09.172299 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.172198    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 09:35:09.172378 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.172315    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gztjj\""
Nov 02 09:35:09.172590 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.172566    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-trusted-ca\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.172746 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.172730    2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:09.172815 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.172764    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2608084-c9df-4998-a521-fb2239b3c349-serving-cert\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:09.172815 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.172783    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls podName:3c01c082-8479-460d-a80d-5e590ab2fcbe nodeName:}" failed. No retries permitted until 2026-11-02 09:35:09.672769158 +0000 UTC m=+32.784552528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mlxbp" (UID: "3c01c082-8479-460d-a80d-5e590ab2fcbe") : secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:09.172903 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.172872    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:09.672857956 +0000 UTC m=+32.784641385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : configmap references non-existent config key: service-ca.crt
Nov 02 09:35:09.173868 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.173710    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d28887e-4aa9-4209-a51a-fec831136660-trusted-ca\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:09.173868 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.173815    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xtlc6"
Nov 02 09:35:09.174062 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.174038    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2608084-c9df-4998-a521-fb2239b3c349-config\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:09.174154 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.174086    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-image-registry-private-configuration\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.174345 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.174321    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-default-certificate\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.174675 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.174657    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-installation-pull-secrets\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.175369 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.175331    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d28887e-4aa9-4209-a51a-fec831136660-serving-cert\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:09.176620 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.176597    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-stats-auth\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.184556 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.184529    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzrxx\" (UniqueName: \"kubernetes.io/projected/a2608084-c9df-4998-a521-fb2239b3c349-kube-api-access-zzrxx\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:09.184914 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.184886    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4zq4\" (UniqueName: \"kubernetes.io/projected/80746de3-f9f3-4efe-a963-9b37f290fcb7-kube-api-access-q4zq4\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.185626 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.185607    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-bound-sa-token\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.185911 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.185893    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8ms9\" (UniqueName: \"kubernetes.io/projected/2d28887e-4aa9-4209-a51a-fec831136660-kube-api-access-n8ms9\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:09.185974 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.185911    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhsmx\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-kube-api-access-xhsmx\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.186186 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.186172    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b2xg\" (UniqueName: \"kubernetes.io/projected/3c01c082-8479-460d-a80d-5e590ab2fcbe-kube-api-access-7b2xg\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:09.222884 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.222754    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:09.264083 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.264047    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:09.272080 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272046    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmw7c\" (UniqueName: \"kubernetes.io/projected/b9ed0d41-1122-4ec2-a9af-e84cc71605df-kube-api-access-pmw7c\") pod \"volume-data-source-validator-58c9f5f46d-sb5ns\" (UID: \"b9ed0d41-1122-4ec2-a9af-e84cc71605df\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sb5ns"
Nov 02 09:35:09.272190 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272096    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6kp\" (UniqueName: \"kubernetes.io/projected/26adcc96-ef0b-416f-8b15-e01be59fb5ae-kube-api-access-zr6kp\") pod \"ingress-canary-xkrcs\" (UID: \"26adcc96-ef0b-416f-8b15-e01be59fb5ae\") " pod="openshift-ingress-canary/ingress-canary-xkrcs"
Nov 02 09:35:09.272190 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272127    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:09.272190 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272165    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e5393080-e20a-47fc-8dc7-125f42457736-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:09.272380 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272205    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgmvv\" (UniqueName: \"kubernetes.io/projected/502f9765-e116-4359-b513-c1d00c90d7de-kube-api-access-jgmvv\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:09.272380 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272232    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26adcc96-ef0b-416f-8b15-e01be59fb5ae-cert\") pod \"ingress-canary-xkrcs\" (UID: \"26adcc96-ef0b-416f-8b15-e01be59fb5ae\") " pod="openshift-ingress-canary/ingress-canary-xkrcs"
Nov 02 09:35:09.272380 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272260    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/502f9765-e116-4359-b513-c1d00c90d7de-config-volume\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:09.272380 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272321    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbd4d998-a5be-4a0a-9113-8f3782df753a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:09.272380 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272374    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:09.272624 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272428    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:09.272624 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272456    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:09.272624 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272498    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8htqg\" (UniqueName: \"kubernetes.io/projected/a05b1aec-cdf3-424b-8dfe-d4a6f06d6536-kube-api-access-8htqg\") pod \"network-check-source-6f78d59f99-mvxdw\" (UID: \"a05b1aec-cdf3-424b-8dfe-d4a6f06d6536\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvxdw"
Nov 02 09:35:09.272624 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272539    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86xb\" (UniqueName: \"kubernetes.io/projected/dbd4d998-a5be-4a0a-9113-8f3782df753a-kube-api-access-b86xb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:09.272624 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272563    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502f9765-e116-4359-b513-c1d00c90d7de-tmp-dir\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:09.272868 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272628    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbd4d998-a5be-4a0a-9113-8f3782df753a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:09.272868 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.272669    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbgj\" (UniqueName: \"kubernetes.io/projected/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-kube-api-access-zsbgj\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:09.273588 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.273552    2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 09:35:09.273684 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.273624    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert podName:e5393080-e20a-47fc-8dc7-125f42457736 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:09.773604967 +0000 UTC m=+32.885388331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qrb6h" (UID: "e5393080-e20a-47fc-8dc7-125f42457736") : secret "networking-console-plugin-cert" not found
Nov 02 09:35:09.275576 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.274720    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e5393080-e20a-47fc-8dc7-125f42457736-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:09.277376 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.275706    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbd4d998-a5be-4a0a-9113-8f3782df753a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:09.277376 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.275978    2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 09:35:09.277376 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.276026    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls podName:5e2f49f4-2884-4804-bb3a-3ee463eeb4f8 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:09.776010562 +0000 UTC m=+32.887793923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ntl2c" (UID: "5e2f49f4-2884-4804-bb3a-3ee463eeb4f8") : secret "samples-operator-tls" not found
Nov 02 09:35:09.278928 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.278508    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:09.279926 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.279266    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26adcc96-ef0b-416f-8b15-e01be59fb5ae-cert\") pod \"ingress-canary-xkrcs\" (UID: \"26adcc96-ef0b-416f-8b15-e01be59fb5ae\") " pod="openshift-ingress-canary/ingress-canary-xkrcs"
Nov 02 09:35:09.282061 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.281692    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbd4d998-a5be-4a0a-9113-8f3782df753a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:09.286121 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.286092    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbgj\" (UniqueName: \"kubernetes.io/projected/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-kube-api-access-zsbgj\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:09.289179 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.289133    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8htqg\" (UniqueName: \"kubernetes.io/projected/a05b1aec-cdf3-424b-8dfe-d4a6f06d6536-kube-api-access-8htqg\") pod \"network-check-source-6f78d59f99-mvxdw\" (UID: \"a05b1aec-cdf3-424b-8dfe-d4a6f06d6536\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvxdw"
Nov 02 09:35:09.293977 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.291784    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6kp\" (UniqueName: \"kubernetes.io/projected/26adcc96-ef0b-416f-8b15-e01be59fb5ae-kube-api-access-zr6kp\") pod \"ingress-canary-xkrcs\" (UID: \"26adcc96-ef0b-416f-8b15-e01be59fb5ae\") " pod="openshift-ingress-canary/ingress-canary-xkrcs"
Nov 02 09:35:09.293977 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.293782    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86xb\" (UniqueName: \"kubernetes.io/projected/dbd4d998-a5be-4a0a-9113-8f3782df753a-kube-api-access-b86xb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:09.294232 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.294211    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmw7c\" (UniqueName: \"kubernetes.io/projected/b9ed0d41-1122-4ec2-a9af-e84cc71605df-kube-api-access-pmw7c\") pod \"volume-data-source-validator-58c9f5f46d-sb5ns\" (UID: \"b9ed0d41-1122-4ec2-a9af-e84cc71605df\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sb5ns"
Nov 02 09:35:09.305846 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.305562    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:09.342862 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.342820    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sb5ns"
Nov 02 09:35:09.362644 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.359039    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:09.376081 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.374653    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgmvv\" (UniqueName: \"kubernetes.io/projected/502f9765-e116-4359-b513-c1d00c90d7de-kube-api-access-jgmvv\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:09.376081 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.374698    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/502f9765-e116-4359-b513-c1d00c90d7de-config-volume\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:09.376081 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.374797    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:09.376081 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.374844    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502f9765-e116-4359-b513-c1d00c90d7de-tmp-dir\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:09.376081 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.375205    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502f9765-e116-4359-b513-c1d00c90d7de-tmp-dir\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:09.376081 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.375629    2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:09.376081 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.375680    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls podName:502f9765-e116-4359-b513-c1d00c90d7de nodeName:}" failed. No retries permitted until 2026-11-02 09:35:09.875667494 +0000 UTC m=+32.987450852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls") pod "dns-default-zgjvs" (UID: "502f9765-e116-4359-b513-c1d00c90d7de") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:09.376081 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.375799    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/502f9765-e116-4359-b513-c1d00c90d7de-config-volume\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:09.398165 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.398140    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgmvv\" (UniqueName: \"kubernetes.io/projected/502f9765-e116-4359-b513-c1d00c90d7de-kube-api-access-jgmvv\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:09.436874 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.436192    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:35:09.436874 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.436192    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:09.436874 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.436817    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:09.438761 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.438738    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvxdw"
Nov 02 09:35:09.442661 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.442202    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 09:35:09.442661 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.442336    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sbddj\""
Nov 02 09:35:09.442661 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.442418    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 09:35:09.442661 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.442517    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbhf4\""
Nov 02 09:35:09.442661 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.442621    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9d4cl\""
Nov 02 09:35:09.460593 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.460578    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xkrcs"
Nov 02 09:35:09.491869 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.491848    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xtlc6"]
Nov 02 09:35:09.495269 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.495173    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cf7t5"]
Nov 02 09:35:09.498446 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:35:09.498015    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce2663d_12a8_427f_b8ec_0d2238a06470.slice/crio-ae2122bcae7d253da6f1dbaca0bd3c1a99f8f8fe90d1da4ab674d862b67ae07e WatchSource:0}: Error finding container ae2122bcae7d253da6f1dbaca0bd3c1a99f8f8fe90d1da4ab674d862b67ae07e: Status 404 returned error can't find the container with id ae2122bcae7d253da6f1dbaca0bd3c1a99f8f8fe90d1da4ab674d862b67ae07e
Nov 02 09:35:09.498969 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:35:09.498949    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78cde7e0_a2f3_4f33_ba88_6c9fedc723eb.slice/crio-e53194c8e8afd2068f00b559903228fed6cb73e0d4677b13dac8803b9f0fe2c3 WatchSource:0}: Error finding container e53194c8e8afd2068f00b559903228fed6cb73e0d4677b13dac8803b9f0fe2c3: Status 404 returned error can't find the container with id e53194c8e8afd2068f00b559903228fed6cb73e0d4677b13dac8803b9f0fe2c3
Nov 02 09:35:09.499431 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.499129    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pprd9"]
Nov 02 09:35:09.502043 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:35:09.501807    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d7a9df_37e3_492b_9540_3df603931723.slice/crio-f5451ba8c922662da7e07f82608496e5dad034111e57153ecbe5c8bce1d22f64 WatchSource:0}: Error finding container f5451ba8c922662da7e07f82608496e5dad034111e57153ecbe5c8bce1d22f64: Status 404 returned error can't find the container with id f5451ba8c922662da7e07f82608496e5dad034111e57153ecbe5c8bce1d22f64
Nov 02 09:35:09.559398 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.558275    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-pdw8n"]
Nov 02 09:35:09.574999 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.574281    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"]
Nov 02 09:35:09.577091 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.577068    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:09.577243 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.577227    2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:09.577305 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.577246    2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8fbd47b-96z8h: secret "image-registry-tls" not found
Nov 02 09:35:09.577305 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.577296    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls podName:2daf8a59-8027-42b4-8414-55a0883bc04f nodeName:}" failed. No retries permitted until 2026-11-02 09:35:10.577278465 +0000 UTC m=+33.689061840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls") pod "image-registry-bc8fbd47b-96z8h" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f") : secret "image-registry-tls" not found
Nov 02 09:35:09.581173 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:35:09.581148    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2608084_c9df_4998_a521_fb2239b3c349.slice/crio-24d082cb9734f62d614ce4fab957d985b65f9124bb800317b72a70da745147f2 WatchSource:0}: Error finding container 24d082cb9734f62d614ce4fab957d985b65f9124bb800317b72a70da745147f2: Status 404 returned error can't find the container with id 24d082cb9734f62d614ce4fab957d985b65f9124bb800317b72a70da745147f2
Nov 02 09:35:09.612689 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.612615    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-n99gd"]
Nov 02 09:35:09.644752 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.644725    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sb5ns"]
Nov 02 09:35:09.647219 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:35:09.647191    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ed0d41_1122_4ec2_a9af_e84cc71605df.slice/crio-e35a99be9186f5ac6d00d0ae645a12ab8c5a3aa4dafcf784eaadb76c61aa057c WatchSource:0}: Error finding container e35a99be9186f5ac6d00d0ae645a12ab8c5a3aa4dafcf784eaadb76c61aa057c: Status 404 returned error can't find the container with id e35a99be9186f5ac6d00d0ae645a12ab8c5a3aa4dafcf784eaadb76c61aa057c
Nov 02 09:35:09.656071 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.656004    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"]
Nov 02 09:35:09.678818 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.678387    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:09.678818 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.678427    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.678818 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.678516    2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:09.678818 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.678527    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:09.678818 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.678564    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:09.678818 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.678630    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls podName:3c01c082-8479-460d-a80d-5e590ab2fcbe nodeName:}" failed. No retries permitted until 2026-11-02 09:35:10.678613041 +0000 UTC m=+33.790396402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mlxbp" (UID: "3c01c082-8479-460d-a80d-5e590ab2fcbe") : secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:09.678818 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.678710    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:10.678701052 +0000 UTC m=+33.790484438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : configmap references non-existent config key: service-ca.crt
Nov 02 09:35:09.678818 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.678750    2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:09.678818 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.678764    2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79578858d5-6w82j: secret "image-registry-tls" not found
Nov 02 09:35:09.678818 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.678774    2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 09:35:09.678818 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.678804    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls podName:3603dee5-a95f-47e4-9d60-381e82b8afa4 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:10.678787705 +0000 UTC m=+33.790571070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls") pod "image-registry-79578858d5-6w82j" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4") : secret "image-registry-tls" not found
Nov 02 09:35:09.678818 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.678821    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:10.678812852 +0000 UTC m=+33.790596213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : secret "router-metrics-certs-default" not found
Nov 02 09:35:09.693949 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.693922    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd" event={"ID":"dbd4d998-a5be-4a0a-9113-8f3782df753a","Type":"ContainerStarted","Data":"cb423a8e648d75df0e1c33716b4e847e2a8c486e2b4214692de72b015b184944"}
Nov 02 09:35:09.695045 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.695027    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sb5ns" event={"ID":"b9ed0d41-1122-4ec2-a9af-e84cc71605df","Type":"ContainerStarted","Data":"e35a99be9186f5ac6d00d0ae645a12ab8c5a3aa4dafcf784eaadb76c61aa057c"}
Nov 02 09:35:09.695939 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.695923    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xtlc6" event={"ID":"fce2663d-12a8-427f-b8ec-0d2238a06470","Type":"ContainerStarted","Data":"ae2122bcae7d253da6f1dbaca0bd3c1a99f8f8fe90d1da4ab674d862b67ae07e"}
Nov 02 09:35:09.696927 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.696900    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cf7t5" event={"ID":"78cde7e0-a2f3-4f33-ba88-6c9fedc723eb","Type":"ContainerStarted","Data":"e53194c8e8afd2068f00b559903228fed6cb73e0d4677b13dac8803b9f0fe2c3"}
Nov 02 09:35:09.697888 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.697871    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pprd9" event={"ID":"95d7a9df-37e3-492b-9540-3df603931723","Type":"ContainerStarted","Data":"f5451ba8c922662da7e07f82608496e5dad034111e57153ecbe5c8bce1d22f64"}
Nov 02 09:35:09.698883 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.698867    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw" event={"ID":"a2608084-c9df-4998-a521-fb2239b3c349","Type":"ContainerStarted","Data":"24d082cb9734f62d614ce4fab957d985b65f9124bb800317b72a70da745147f2"}
Nov 02 09:35:09.699951 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.699935    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-pdw8n" event={"ID":"85741a00-b702-46cb-a90c-e000bf9976f0","Type":"ContainerStarted","Data":"94ae0eb45814cb611198f1782e1a3ff3853868f6846a90ee67c74e4dfdcdffe4"}
Nov 02 09:35:09.701696 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.701672    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd" event={"ID":"2d28887e-4aa9-4209-a51a-fec831136660","Type":"ContainerStarted","Data":"878c16cc78814b967d23734afb4160c22728b32027f4a65d65d24aca63ebcd6b"}
Nov 02 09:35:09.742218 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.742195    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-mvxdw"]
Nov 02 09:35:09.745105 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:35:09.745074    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05b1aec_cdf3_424b_8dfe_d4a6f06d6536.slice/crio-2f3f8a01c0a8c37fc568a03f5687d6ee6ab5d9f8dd8c0d5ff4ca053f30d7aeb5 WatchSource:0}: Error finding container 2f3f8a01c0a8c37fc568a03f5687d6ee6ab5d9f8dd8c0d5ff4ca053f30d7aeb5: Status 404 returned error can't find the container with id 2f3f8a01c0a8c37fc568a03f5687d6ee6ab5d9f8dd8c0d5ff4ca053f30d7aeb5
Nov 02 09:35:09.770023 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.769998    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xkrcs"]
Nov 02 09:35:09.772633 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:35:09.772611    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26adcc96_ef0b_416f_8b15_e01be59fb5ae.slice/crio-48a3b7e17d2c50cb55a8dac745b6af94c7bd3fa462c11fda0d38cb0e30daad72 WatchSource:0}: Error finding container 48a3b7e17d2c50cb55a8dac745b6af94c7bd3fa462c11fda0d38cb0e30daad72: Status 404 returned error can't find the container with id 48a3b7e17d2c50cb55a8dac745b6af94c7bd3fa462c11fda0d38cb0e30daad72
Nov 02 09:35:09.779085 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.779063    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:09.779163 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.779152    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:09.779210 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.779194    2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 09:35:09.779250 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.779244    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert podName:e5393080-e20a-47fc-8dc7-125f42457736 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:10.779230273 +0000 UTC m=+33.891013647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qrb6h" (UID: "e5393080-e20a-47fc-8dc7-125f42457736") : secret "networking-console-plugin-cert" not found
Nov 02 09:35:09.779302 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.779288    2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 09:35:09.779341 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.779332    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls podName:5e2f49f4-2884-4804-bb3a-3ee463eeb4f8 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:10.77932166 +0000 UTC m=+33.891105047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ntl2c" (UID: "5e2f49f4-2884-4804-bb3a-3ee463eeb4f8") : secret "samples-operator-tls" not found
Nov 02 09:35:09.880709 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:09.880530    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:09.880854 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.880683    2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:09.880854 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:09.880833    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls podName:502f9765-e116-4359-b513-c1d00c90d7de nodeName:}" failed. No retries permitted until 2026-11-02 09:35:10.880819169 +0000 UTC m=+33.992602545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls") pod "dns-default-zgjvs" (UID: "502f9765-e116-4359-b513-c1d00c90d7de") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:10.183460 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.183366    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:10.183616 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.183467    2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 09:35:10.183616 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.183518    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:10.183616 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.183550    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs podName:961f499a-2ed3-4e0d-b5d2-9194d18b696a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:42.183518584 +0000 UTC m=+65.295301962 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs") pod "network-metrics-daemon-v9xng" (UID: "961f499a-2ed3-4e0d-b5d2-9194d18b696a") : secret "metrics-daemon-secret" not found
Nov 02 09:35:10.186366 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.186327    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:10.285092 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.285050    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfq4v\" (UniqueName: \"kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v\") pod \"network-check-target-h5bzx\" (UID: \"dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e\") " pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:35:10.287936 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.287911    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfq4v\" (UniqueName: \"kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v\") pod \"network-check-target-h5bzx\" (UID: \"dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e\") " pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:35:10.376242 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.376208    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:10.399442 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.399409    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:35:10.589218 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.588518    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:10.589218 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.588796    2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:10.589218 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.588813    2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8fbd47b-96z8h: secret "image-registry-tls" not found
Nov 02 09:35:10.589218 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.588870    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls podName:2daf8a59-8027-42b4-8414-55a0883bc04f nodeName:}" failed. No retries permitted until 2026-11-02 09:35:12.58885076 +0000 UTC m=+35.700634134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls") pod "image-registry-bc8fbd47b-96z8h" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f") : secret "image-registry-tls" not found
Nov 02 09:35:10.679745 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.679713    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lxn8z"]
Nov 02 09:35:10.688727 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:35:10.688695    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b95e7b_2ce0_45a1_9859_0d2378576064.slice/crio-4b5684e60afbf8fd4dbca78f242bb6968081d37cea6f1a5529aa2c9fca61202f WatchSource:0}: Error finding container 4b5684e60afbf8fd4dbca78f242bb6968081d37cea6f1a5529aa2c9fca61202f: Status 404 returned error can't find the container with id 4b5684e60afbf8fd4dbca78f242bb6968081d37cea6f1a5529aa2c9fca61202f
Nov 02 09:35:10.689083 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.689059    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:10.689175 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.689106    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:10.689233 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.689193    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:10.689233 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.689222    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:10.689455 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.689434    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:12.689414227 +0000 UTC m=+35.801197611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : configmap references non-existent config key: service-ca.crt
Nov 02 09:35:10.689865 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.689846    2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:10.689939 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.689868    2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79578858d5-6w82j: secret "image-registry-tls" not found
Nov 02 09:35:10.689939 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.689915    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls podName:3603dee5-a95f-47e4-9d60-381e82b8afa4 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:12.689899936 +0000 UTC m=+35.801683319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls") pod "image-registry-79578858d5-6w82j" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4") : secret "image-registry-tls" not found
Nov 02 09:35:10.690044 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.689976    2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 09:35:10.690044 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.690007    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:12.689996781 +0000 UTC m=+35.801780154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : secret "router-metrics-certs-default" not found
Nov 02 09:35:10.690151 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.690057    2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:10.690151 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.690087    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls podName:3c01c082-8479-460d-a80d-5e590ab2fcbe nodeName:}" failed. No retries permitted until 2026-11-02 09:35:12.690077768 +0000 UTC m=+35.801861127 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mlxbp" (UID: "3c01c082-8479-460d-a80d-5e590ab2fcbe") : secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:10.709294 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.709108    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xtlc6" event={"ID":"fce2663d-12a8-427f-b8ec-0d2238a06470","Type":"ContainerStarted","Data":"77b5f0ac6ba095bfaf6f998c8020db70d5a7617ef8ebd4f9654763a3cb55c876"}
Nov 02 09:35:10.723757 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.722736    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h5bzx"]
Nov 02 09:35:10.723757 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.722948    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cf7t5" event={"ID":"78cde7e0-a2f3-4f33-ba88-6c9fedc723eb","Type":"ContainerStarted","Data":"ebabc54b82cb6ffaa6dd5c0a11810c05de8fc1b7ac6e89bf3509f62edd94206e"}
Nov 02 09:35:10.726726 ip-10-0-134-174 kubenswrapper[2545]: W1102 09:35:10.726695    2545 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd65a5e_d4eb_4e5a_8a99_e3431b306c4e.slice/crio-25fadf7ae5035e6ea77cbd198cd7511ff7ac7c0db5a76eb943bab1dd1f8c7319 WatchSource:0}: Error finding container 25fadf7ae5035e6ea77cbd198cd7511ff7ac7c0db5a76eb943bab1dd1f8c7319: Status 404 returned error can't find the container with id 25fadf7ae5035e6ea77cbd198cd7511ff7ac7c0db5a76eb943bab1dd1f8c7319
Nov 02 09:35:10.731052 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.731025    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pprd9" event={"ID":"95d7a9df-37e3-492b-9540-3df603931723","Type":"ContainerStarted","Data":"ef1a81ed6f6a65144adb9fd69fce112d7876056c79ae507adebd290779c3147d"}
Nov 02 09:35:10.734128 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.734108    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xkrcs" event={"ID":"26adcc96-ef0b-416f-8b15-e01be59fb5ae","Type":"ContainerStarted","Data":"48a3b7e17d2c50cb55a8dac745b6af94c7bd3fa462c11fda0d38cb0e30daad72"}
Nov 02 09:35:10.735784 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.735729    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvxdw" event={"ID":"a05b1aec-cdf3-424b-8dfe-d4a6f06d6536","Type":"ContainerStarted","Data":"2f3f8a01c0a8c37fc568a03f5687d6ee6ab5d9f8dd8c0d5ff4ca053f30d7aeb5"}
Nov 02 09:35:10.737344 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.737314    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lxn8z" event={"ID":"c5b95e7b-2ce0-45a1-9859-0d2378576064","Type":"ContainerStarted","Data":"4b5684e60afbf8fd4dbca78f242bb6968081d37cea6f1a5529aa2c9fca61202f"}
Nov 02 09:35:10.747648 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.746378    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-cf7t5" podStartSLOduration=413.746348044 podStartE2EDuration="6m53.746348044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:10.745706494 +0000 UTC m=+33.857489872" watchObservedRunningTime="2026-11-02 09:35:10.746348044 +0000 UTC m=+33.858131428"
Nov 02 09:35:10.747648 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.747038    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xtlc6" podStartSLOduration=413.747029252 podStartE2EDuration="6m53.747029252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:10.726287977 +0000 UTC m=+33.838071358" watchObservedRunningTime="2026-11-02 09:35:10.747029252 +0000 UTC m=+33.858812634"
Nov 02 09:35:10.771287 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.770914    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-pprd9" podStartSLOduration=413.770898294 podStartE2EDuration="6m53.770898294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:10.7681011 +0000 UTC m=+33.879884478" watchObservedRunningTime="2026-11-02 09:35:10.770898294 +0000 UTC m=+33.882681678"
Nov 02 09:35:10.792616 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.790608    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:10.792616 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.790761    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:10.792616 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.791686    2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 09:35:10.792616 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.791746    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert podName:e5393080-e20a-47fc-8dc7-125f42457736 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:12.791726553 +0000 UTC m=+35.903509927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qrb6h" (UID: "e5393080-e20a-47fc-8dc7-125f42457736") : secret "networking-console-plugin-cert" not found
Nov 02 09:35:10.792616 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.792452    2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 09:35:10.792616 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.792496    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls podName:5e2f49f4-2884-4804-bb3a-3ee463eeb4f8 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:12.792483622 +0000 UTC m=+35.904266984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ntl2c" (UID: "5e2f49f4-2884-4804-bb3a-3ee463eeb4f8") : secret "samples-operator-tls" not found
Nov 02 09:35:10.892979 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:10.892270    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:10.892979 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.892446    2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:10.892979 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:10.892552    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls podName:502f9765-e116-4359-b513-c1d00c90d7de nodeName:}" failed. No retries permitted until 2026-11-02 09:35:12.892534489 +0000 UTC m=+36.004317867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls") pod "dns-default-zgjvs" (UID: "502f9765-e116-4359-b513-c1d00c90d7de") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:11.750395 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:11.750337    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h5bzx" event={"ID":"dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e","Type":"ContainerStarted","Data":"25fadf7ae5035e6ea77cbd198cd7511ff7ac7c0db5a76eb943bab1dd1f8c7319"}
Nov 02 09:35:12.611954 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:12.611471    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:12.611954 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.611612    2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:12.611954 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.611625    2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8fbd47b-96z8h: secret "image-registry-tls" not found
Nov 02 09:35:12.611954 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.611682    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls podName:2daf8a59-8027-42b4-8414-55a0883bc04f nodeName:}" failed. No retries permitted until 2026-11-02 09:35:16.611670192 +0000 UTC m=+39.723453565 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls") pod "image-registry-bc8fbd47b-96z8h" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f") : secret "image-registry-tls" not found
Nov 02 09:35:12.712197 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:12.712164    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:12.712197 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:12.712201    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:12.712422 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:12.712253    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:12.712422 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:12.712282    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:12.712502 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.712466    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:16.712446888 +0000 UTC m=+39.824230262 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : configmap references non-existent config key: service-ca.crt
Nov 02 09:35:12.712557 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.712532    2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:12.712557 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.712541    2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79578858d5-6w82j: secret "image-registry-tls" not found
Nov 02 09:35:12.712636 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.712564    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls podName:3603dee5-a95f-47e4-9d60-381e82b8afa4 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:16.71255756 +0000 UTC m=+39.824340932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls") pod "image-registry-79578858d5-6w82j" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4") : secret "image-registry-tls" not found
Nov 02 09:35:12.712636 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.712604    2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 09:35:12.712636 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.712622    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:16.712616329 +0000 UTC m=+39.824399700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : secret "router-metrics-certs-default" not found
Nov 02 09:35:12.712747 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.712654    2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:12.712747 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.712673    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls podName:3c01c082-8479-460d-a80d-5e590ab2fcbe nodeName:}" failed. No retries permitted until 2026-11-02 09:35:16.712667944 +0000 UTC m=+39.824451302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mlxbp" (UID: "3c01c082-8479-460d-a80d-5e590ab2fcbe") : secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:12.761249 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:12.761210    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sb5ns" event={"ID":"b9ed0d41-1122-4ec2-a9af-e84cc71605df","Type":"ContainerStarted","Data":"1c09ee1c8220c1215726b4d2a073bc146da27d6ff7b0853b81fad9b6341f9be2"}
Nov 02 09:35:12.813715 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:12.813006    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:12.813715 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:12.813144    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:12.819094 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.818513    2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 09:35:12.819094 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.818597    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert podName:e5393080-e20a-47fc-8dc7-125f42457736 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:16.818575497 +0000 UTC m=+39.930358870 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qrb6h" (UID: "e5393080-e20a-47fc-8dc7-125f42457736") : secret "networking-console-plugin-cert" not found
Nov 02 09:35:12.821314 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.819527    2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 09:35:12.821314 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.819596    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls podName:5e2f49f4-2884-4804-bb3a-3ee463eeb4f8 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:16.819576982 +0000 UTC m=+39.931360354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ntl2c" (UID: "5e2f49f4-2884-4804-bb3a-3ee463eeb4f8") : secret "samples-operator-tls" not found
Nov 02 09:35:12.914725 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:12.914001    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:12.914725 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.914213    2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:12.914725 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:12.914276    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls podName:502f9765-e116-4359-b513-c1d00c90d7de nodeName:}" failed. No retries permitted until 2026-11-02 09:35:16.914258814 +0000 UTC m=+40.026042187 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls") pod "dns-default-zgjvs" (UID: "502f9765-e116-4359-b513-c1d00c90d7de") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:13.786328 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:13.785026    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sb5ns" podStartSLOduration=385.846352191 podStartE2EDuration="6m28.785006966s" totalImagesPullingTime="2.938654775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:13.78461233 +0000 UTC m=+36.896395707" watchObservedRunningTime="2026-11-02 09:35:13.785006966 +0000 UTC m=+36.896790347"
Nov 02 09:35:15.778107 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:15.776739    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd" event={"ID":"dbd4d998-a5be-4a0a-9113-8f3782df753a","Type":"ContainerStarted","Data":"f0b11800a2ea74eb653c2b80deda00c5a253da6dfb8f06203a70d562d17991b6"}
Nov 02 09:35:15.780467 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:15.780313    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-pdw8n" event={"ID":"85741a00-b702-46cb-a90c-e000bf9976f0","Type":"ContainerStarted","Data":"c6f9f6589c6fd87152a37df3f40c37fcd34c20735131ab3e8ee4064de74a8f07"}
Nov 02 09:35:15.784218 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:15.784195    2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/0.log"
Nov 02 09:35:15.784365 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:15.784231    2545 generic.go:356] "Generic (PLEG): container finished" podID="2d28887e-4aa9-4209-a51a-fec831136660" containerID="67c67e454f7c68d1b28e62be1ed51e2e8e6c5ade0e033435a85063caa9850613" exitCode=255
Nov 02 09:35:15.784365 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:15.784283    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd" event={"ID":"2d28887e-4aa9-4209-a51a-fec831136660","Type":"ContainerDied","Data":"67c67e454f7c68d1b28e62be1ed51e2e8e6c5ade0e033435a85063caa9850613"}
Nov 02 09:35:15.784823 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:15.784569    2545 scope.go:122] "RemoveContainer" containerID="67c67e454f7c68d1b28e62be1ed51e2e8e6c5ade0e033435a85063caa9850613"
Nov 02 09:35:15.803985 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:15.801341    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd" podStartSLOduration=385.079007326 podStartE2EDuration="6m30.801324209s" totalImagesPullingTime="5.722316883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:15.799747773 +0000 UTC m=+38.911531154" watchObservedRunningTime="2026-11-02 09:35:15.801324209 +0000 UTC m=+38.913107586"
Nov 02 09:35:16.418001 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.417744    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-pdw8n" podStartSLOduration=385.56836421 podStartE2EDuration="6m31.417724059s" totalImagesPullingTime="5.849359849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:15.84533138 +0000 UTC m=+38.957114761" watchObservedRunningTime="2026-11-02 09:35:16.417724059 +0000 UTC m=+39.529507438"
Nov 02 09:35:16.418416 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.418395    2545 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nbfgg"]
Nov 02 09:35:16.448031 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.448003    2545 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nbfgg"]
Nov 02 09:35:16.448314 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.448290    2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:16.457508 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.455901    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 09:35:16.457508 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.456106    2545 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xrsh4\""
Nov 02 09:35:16.457508 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.456180    2545 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 09:35:16.557237 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.557112    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9tb2\" (UniqueName: \"kubernetes.io/projected/b36bfac6-a600-4029-a446-7de82575fb0a-kube-api-access-n9tb2\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:16.557237 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.557169    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:16.557237 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.557234    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b36bfac6-a600-4029-a446-7de82575fb0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:16.558101 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.557310    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b36bfac6-a600-4029-a446-7de82575fb0a-crio-socket\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:16.558101 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.557388    2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b36bfac6-a600-4029-a446-7de82575fb0a-data-volume\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:16.658854 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.658586    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9tb2\" (UniqueName: \"kubernetes.io/projected/b36bfac6-a600-4029-a446-7de82575fb0a-kube-api-access-n9tb2\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:16.658854 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.658655    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:16.658854 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.658726    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b36bfac6-a600-4029-a446-7de82575fb0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:16.659206 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.659185    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:16.659286 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.659271    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b36bfac6-a600-4029-a446-7de82575fb0a-crio-socket\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:16.659336 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.659302    2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 09:35:16.659397 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.659346    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b36bfac6-a600-4029-a446-7de82575fb0a-data-volume\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:16.659456 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.659396    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b36bfac6-a600-4029-a446-7de82575fb0a-kube-rbac-proxy-cm podName:b36bfac6-a600-4029-a446-7de82575fb0a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:17.15937453 +0000 UTC m=+40.271157902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b36bfac6-a600-4029-a446-7de82575fb0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nbfgg" (UID: "b36bfac6-a600-4029-a446-7de82575fb0a") : configmap "kube-rbac-proxy" not found
Nov 02 09:35:16.659907 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.659635    2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:16.659907 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.659652    2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8fbd47b-96z8h: secret "image-registry-tls" not found
Nov 02 09:35:16.659907 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.659694    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls podName:2daf8a59-8027-42b4-8414-55a0883bc04f nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.659679707 +0000 UTC m=+47.771463070 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls") pod "image-registry-bc8fbd47b-96z8h" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f") : secret "image-registry-tls" not found
Nov 02 09:35:16.659907 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.659746    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b36bfac6-a600-4029-a446-7de82575fb0a-data-volume\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:16.659907 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.659839    2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:16.659907 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.659868    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b36bfac6-a600-4029-a446-7de82575fb0a-crio-socket\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:16.659907 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.659905    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls podName:b36bfac6-a600-4029-a446-7de82575fb0a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:17.159888003 +0000 UTC m=+40.271671376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nbfgg" (UID: "b36bfac6-a600-4029-a446-7de82575fb0a") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:16.677251 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.677221    2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9tb2\" (UniqueName: \"kubernetes.io/projected/b36bfac6-a600-4029-a446-7de82575fb0a-kube-api-access-n9tb2\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:16.760158 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.760121    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:16.760343 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.760177    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:16.760343 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.760220    2545 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:16.760343 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.760285    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls podName:3c01c082-8479-460d-a80d-5e590ab2fcbe nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.760267412 +0000 UTC m=+47.872050794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mlxbp" (UID: "3c01c082-8479-460d-a80d-5e590ab2fcbe") : secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:16.760343 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.760328    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.760318405 +0000 UTC m=+47.872101772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : configmap references non-existent config key: service-ca.crt
Nov 02 09:35:16.760577 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.760344    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:16.760577 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.760507    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:16.760577 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.760456    2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:16.760712 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.760586    2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79578858d5-6w82j: secret "image-registry-tls" not found
Nov 02 09:35:16.760712 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.760613    2545 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 09:35:16.760712 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.760666    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.760648969 +0000 UTC m=+47.872432337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : secret "router-metrics-certs-default" not found
Nov 02 09:35:16.760712 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.760685    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls podName:3603dee5-a95f-47e4-9d60-381e82b8afa4 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.760675931 +0000 UTC m=+47.872459304 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls") pod "image-registry-79578858d5-6w82j" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4") : secret "image-registry-tls" not found
Nov 02 09:35:16.795172 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.795079    2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/1.log"
Nov 02 09:35:16.797626 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.797595    2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/0.log"
Nov 02 09:35:16.797772 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.797640    2545 generic.go:356] "Generic (PLEG): container finished" podID="2d28887e-4aa9-4209-a51a-fec831136660" containerID="ce9276cd5977d158006454c6d5076ffdb908f3171c697d08603d28e20fa97374" exitCode=255
Nov 02 09:35:16.798520 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.798077    2545 scope.go:122] "RemoveContainer" containerID="ce9276cd5977d158006454c6d5076ffdb908f3171c697d08603d28e20fa97374"
Nov 02 09:35:16.798520 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.798294    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n99gd_openshift-console-operator(2d28887e-4aa9-4209-a51a-fec831136660)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd" podUID="2d28887e-4aa9-4209-a51a-fec831136660"
Nov 02 09:35:16.798968 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.798752    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd" event={"ID":"2d28887e-4aa9-4209-a51a-fec831136660","Type":"ContainerDied","Data":"ce9276cd5977d158006454c6d5076ffdb908f3171c697d08603d28e20fa97374"}
Nov 02 09:35:16.798968 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.798832    2545 scope.go:122] "RemoveContainer" containerID="67c67e454f7c68d1b28e62be1ed51e2e8e6c5ade0e033435a85063caa9850613"
Nov 02 09:35:16.802060 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.801344    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xkrcs" event={"ID":"26adcc96-ef0b-416f-8b15-e01be59fb5ae","Type":"ContainerStarted","Data":"45dee425b9bc95b4224e6b4128e33920189232cc40ff17e0fe47e299609d80ad"}
Nov 02 09:35:16.803941 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.803512    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw" event={"ID":"a2608084-c9df-4998-a521-fb2239b3c349","Type":"ContainerStarted","Data":"d1547b7165a7f0ccd0408d70640dd252dbe0113b894fe60698dec286042ce221"}
Nov 02 09:35:16.861620 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.861584    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:16.861826 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.861717    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:16.862494 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.861909    2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 09:35:16.862494 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.862110    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert podName:e5393080-e20a-47fc-8dc7-125f42457736 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.862090374 +0000 UTC m=+47.973873734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qrb6h" (UID: "e5393080-e20a-47fc-8dc7-125f42457736") : secret "networking-console-plugin-cert" not found
Nov 02 09:35:16.862494 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.862033    2545 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 09:35:16.862494 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.862468    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls podName:5e2f49f4-2884-4804-bb3a-3ee463eeb4f8 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.862450508 +0000 UTC m=+47.974233881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ntl2c" (UID: "5e2f49f4-2884-4804-bb3a-3ee463eeb4f8") : secret "samples-operator-tls" not found
Nov 02 09:35:16.867738 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.867675    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xkrcs" podStartSLOduration=2.287624803 podStartE2EDuration="8.86763984s" totalImagesPullingTime="6.580015037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:16.866214628 +0000 UTC m=+39.977998012" watchObservedRunningTime="2026-11-02 09:35:16.86763984 +0000 UTC m=+39.979423225"
Nov 02 09:35:16.868951 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.868914    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw" podStartSLOduration=385.592848837 podStartE2EDuration="6m31.868900073s" totalImagesPullingTime="6.276051236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:16.847281779 +0000 UTC m=+39.959065160" watchObservedRunningTime="2026-11-02 09:35:16.868900073 +0000 UTC m=+39.980683463"
Nov 02 09:35:16.962418 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:16.962391    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:16.962614 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.962591    2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:16.962668 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:16.962657    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls podName:502f9765-e116-4359-b513-c1d00c90d7de nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.962639592 +0000 UTC m=+48.074422967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls") pod "dns-default-zgjvs" (UID: "502f9765-e116-4359-b513-c1d00c90d7de") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:17.165311 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:17.165271    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:17.165526 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:17.165379    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b36bfac6-a600-4029-a446-7de82575fb0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:17.165591 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:17.165546    2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 09:35:17.165641 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:17.165607    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b36bfac6-a600-4029-a446-7de82575fb0a-kube-rbac-proxy-cm podName:b36bfac6-a600-4029-a446-7de82575fb0a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:18.165588555 +0000 UTC m=+41.277371916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b36bfac6-a600-4029-a446-7de82575fb0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nbfgg" (UID: "b36bfac6-a600-4029-a446-7de82575fb0a") : configmap "kube-rbac-proxy" not found
Nov 02 09:35:17.166024 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:17.166006    2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:17.166081 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:17.166060    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls podName:b36bfac6-a600-4029-a446-7de82575fb0a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:18.166045116 +0000 UTC m=+41.277828475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nbfgg" (UID: "b36bfac6-a600-4029-a446-7de82575fb0a") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:17.809655 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:17.809629    2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/1.log"
Nov 02 09:35:17.810043 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:17.810028    2545 scope.go:122] "RemoveContainer" containerID="ce9276cd5977d158006454c6d5076ffdb908f3171c697d08603d28e20fa97374"
Nov 02 09:35:17.810252 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:17.810234    2545 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n99gd_openshift-console-operator(2d28887e-4aa9-4209-a51a-fec831136660)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd" podUID="2d28887e-4aa9-4209-a51a-fec831136660"
Nov 02 09:35:17.813158 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:17.813131    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvxdw" event={"ID":"a05b1aec-cdf3-424b-8dfe-d4a6f06d6536","Type":"ContainerStarted","Data":"2704df7dc69a60772517060937fc56526679622a48b4f7f2369460c46edda61c"}
Nov 02 09:35:17.848134 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:17.848079    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvxdw" podStartSLOduration=385.662438798 podStartE2EDuration="6m32.848065439s" totalImagesPullingTime="7.185626641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:17.84741393 +0000 UTC m=+40.959197310" watchObservedRunningTime="2026-11-02 09:35:17.848065439 +0000 UTC m=+40.959848819"
Nov 02 09:35:18.175562 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:18.175478    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:18.175562 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:18.175546    2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b36bfac6-a600-4029-a446-7de82575fb0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:18.175753 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:18.175642    2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:18.175753 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:18.175707    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls podName:b36bfac6-a600-4029-a446-7de82575fb0a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:20.175692304 +0000 UTC m=+43.287475667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nbfgg" (UID: "b36bfac6-a600-4029-a446-7de82575fb0a") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:18.175753 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:18.175652    2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 09:35:18.175890 ip-10-0-134-174 kubenswrapper[2545]: E1102 09:35:18.175770    2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b36bfac6-a600-4029-a446-7de82575fb0a-kube-rbac-proxy-cm podName:b36bfac6-a600-4029-a446-7de82575fb0a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:20.175758508 +0000 UTC m=+43.287541867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b36bfac6-a600-4029-a446-7de82575fb0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nbfgg" (UID: "b36bfac6-a600-4029-a446-7de82575fb0a") : configmap "kube-rbac-proxy" not found
Nov 02 09:35:18.817121 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:18.817097    2545 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h5bzx" event={"ID":"dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e","Type":"ContainerStarted","Data":"13d07c8195baddaf64bb2fc515fbb343e17b7b36a61adafaff1b13105ae52c0f"}
Nov 02 09:35:18.837071 ip-10-0-134-174 kubenswrapper[2545]: I1102 09:35:18.835365    2545 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h5bzx" podStartSLOduration=33.83964169 podStartE2EDuration="40.835336303s" totalImagesPullingTime="6.995694613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:18.835229181 +0000 UTC m=+41.947012561" watchObservedRunningTime="2026-11-02 09:35:18.835336303 +0000 UTC m=+41.947119684"
Nov 02 09:35:19.112758 ip-10-0-134-174 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 09:35:19.124870 ip-10-0-134-174 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 09:35:19.125068 ip-10-0-134-174 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 09:35:19.125226 ip-10-0-134-174 systemd[1]: kubelet.service: Consumed 2.951s CPU time.
Nov 02 09:35:19.137062 ip-10-0-134-174 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 09:35:19.227413 ip-10-0-134-174 kubenswrapper[4614]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 09:35:19.227413 ip-10-0-134-174 kubenswrapper[4614]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 09:35:19.227413 ip-10-0-134-174 kubenswrapper[4614]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 09:35:19.227413 ip-10-0-134-174 kubenswrapper[4614]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 09:35:19.230811 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230721    4614 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:19.230811 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230807    4614 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:19.230811 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230811    4614 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:19.230811 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230815    4614 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:19.230811 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230818    4614 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:19.230811 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230821    4614 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:19.230811 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230824    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230827    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230830    4614 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230833    4614 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230835    4614 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230838    4614 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230841    4614 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230844    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230846    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230849    4614 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230851    4614 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230854    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230856    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230859    4614 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230862    4614 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230864    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230866    4614 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230869    4614 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230872    4614 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230874    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:19.231099 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230877    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230879    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230882    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230884    4614 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230887    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230895    4614 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230900    4614 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230904    4614 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230908    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230912    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230916    4614 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230921    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230925    4614 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230930    4614 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230935    4614 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230940    4614 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230944    4614 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230948    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230952    4614 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230956    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:19.231595 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230959    4614 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230961    4614 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230964    4614 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230967    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230971    4614 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230974    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230976    4614 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.230979    4614 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231006    4614 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231012    4614 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231017    4614 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231021    4614 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231024    4614 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231027    4614 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231030    4614 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231033    4614 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231035    4614 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231038    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231041    4614 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231044    4614 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:19.232293 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231046    4614 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231049    4614 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231052    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231054    4614 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231057    4614 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231059    4614 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231062    4614 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231064    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231068    4614 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231072    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231076    4614 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231079    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231082    4614 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231084    4614 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231087    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231090    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231092    4614 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231095    4614 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231098    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:19.232804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231102    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231105    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231108    4614 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231111    4614 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231113    4614 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231117    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231119    4614 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231122    4614 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231125    4614 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231127    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231130    4614 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231133    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231136    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231138    4614 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231141    4614 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231143    4614 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231146    4614 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231148    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231151    4614 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231153    4614 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:19.233264 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231155    4614 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231984    4614 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231992    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231995    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.231998    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232001    4614 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232006    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232011    4614 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232015    4614 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232019    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232023    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232028    4614 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232032    4614 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232034    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232037    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232040    4614 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232042    4614 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232045    4614 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232048    4614 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232050    4614 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:19.233795 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232053    4614 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232055    4614 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232058    4614 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232060    4614 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232063    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232066    4614 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232069    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232071    4614 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232075    4614 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232079    4614 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232082    4614 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232085    4614 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232088    4614 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232090    4614 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232094    4614 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232096    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232099    4614 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232101    4614 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232104    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:19.234282 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232106    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232109    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232112    4614 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232114    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232118    4614 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232123    4614 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232127    4614 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232129    4614 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232132    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232134    4614 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232137    4614 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232140    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232142    4614 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232145    4614 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232147    4614 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232150    4614 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232153    4614 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232156    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232159    4614 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232161    4614 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:19.234758 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232164    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232166    4614 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232170    4614 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232174    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232178    4614 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232183    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232187    4614 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232190    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232192    4614 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232195    4614 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232198    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232200    4614 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232203    4614 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232206    4614 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232208    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232211    4614 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232213    4614 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232216    4614 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232220    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:19.235256 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232222    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232225    4614 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232228    4614 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232230    4614 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232233    4614 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232236    4614 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232238    4614 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232241    4614 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232243    4614 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232246    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232248    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232251    4614 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232253    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232256    4614 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232258    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232261    4614 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232264    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232266    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232269    4614 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:19.235824 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232271    4614 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232274    4614 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232276    4614 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232279    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232281    4614 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232284    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232287    4614 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232289    4614 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232293    4614 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232296    4614 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232830    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232837    4614 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232841    4614 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232844    4614 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232846    4614 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232850    4614 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232853    4614 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232856    4614 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232861    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:19.236331 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232864    4614 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232866    4614 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232869    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232872    4614 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232874    4614 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232877    4614 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232879    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232882    4614 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232886    4614 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232890    4614 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232893    4614 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232896    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232900    4614 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232903    4614 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232905    4614 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232908    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232911    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232914    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232916    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:19.236830 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232919    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232922    4614 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232925    4614 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232927    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232930    4614 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232932    4614 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232935    4614 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232938    4614 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232941    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232943    4614 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232946    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232948    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232950    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232953    4614 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232956    4614 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232958    4614 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232960    4614 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232963    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232966    4614 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232968    4614 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:19.237326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232971    4614 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232973    4614 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232976    4614 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232978    4614 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232981    4614 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232987    4614 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232989    4614 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232992    4614 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232994    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232996    4614 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.232999    4614 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233001    4614 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233004    4614 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233007    4614 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233010    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233012    4614 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233014    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233017    4614 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233020    4614 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233022    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:19.237907 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233025    4614 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233028    4614 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233030    4614 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233033    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233035    4614 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233038    4614 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233040    4614 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233043    4614 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233045    4614 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233048    4614 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233050    4614 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233053    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233056    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233058    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233060    4614 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233063    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233065    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233069    4614 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233072    4614 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:19.238477 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233075    4614 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233077    4614 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233080    4614 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233082    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233085    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233088    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233091    4614 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233093    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233096    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233099    4614 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233101    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233104    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233106    4614 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233109    4614 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233112    4614 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233116    4614 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233118    4614 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233121    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233123    4614 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233219    4614 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233227    4614 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 09:35:19.238950 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233235    4614 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233242    4614 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233248    4614 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233254    4614 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233258    4614 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233262    4614 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233266    4614 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233269    4614 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233272    4614 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233277    4614 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233280    4614 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233283    4614 flags.go:64] FLAG: --cgroup-root=""
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233287    4614 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233290    4614 flags.go:64] FLAG: --client-ca-file=""
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233292    4614 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233295    4614 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233300    4614 flags.go:64] FLAG: --cluster-domain=""
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233303    4614 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233306    4614 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233309    4614 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233312    4614 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233316    4614 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233319    4614 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 09:35:19.239475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233322    4614 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233325    4614 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233328    4614 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233332    4614 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233335    4614 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233338    4614 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233341    4614 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233345    4614 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233360    4614 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233365    4614 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233370    4614 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233375    4614 flags.go:64] FLAG: --enable-server="true"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233379    4614 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233386    4614 flags.go:64] FLAG: --event-burst="100"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233390    4614 flags.go:64] FLAG: --event-qps="50"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233394    4614 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233397    4614 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233400    4614 flags.go:64] FLAG: --eviction-hard=""
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233404    4614 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233407    4614 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233410    4614 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233414    4614 flags.go:64] FLAG: --eviction-soft=""
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233417    4614 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233420    4614 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233423    4614 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 09:35:19.240049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233426    4614 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233429    4614 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233432    4614 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233435    4614 flags.go:64] FLAG: --feature-gates=""
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233439    4614 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233442    4614 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233445    4614 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233448    4614 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233452    4614 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233455    4614 flags.go:64] FLAG: --help="false"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233458    4614 flags.go:64] FLAG: --hostname-override="ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233461    4614 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233464    4614 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233467    4614 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233471    4614 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233474    4614 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233477    4614 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233480    4614 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233483    4614 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233486    4614 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233489    4614 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233492    4614 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233495    4614 flags.go:64] FLAG: --kube-reserved=""
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233498    4614 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 09:35:19.240691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233501    4614 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233504    4614 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233507    4614 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233509    4614 flags.go:64] FLAG: --lock-file=""
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233512    4614 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233515    4614 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233518    4614 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233524    4614 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233527    4614 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233529    4614 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233532    4614 flags.go:64] FLAG: --logging-format="text"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233535    4614 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233539    4614 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233541    4614 flags.go:64] FLAG: --manifest-url=""
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233544    4614 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233549    4614 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233552    4614 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233555    4614 flags.go:64] FLAG: --max-pods="110"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233558    4614 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233561    4614 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233565    4614 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233568    4614 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233573    4614 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233576    4614 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233579    4614 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 09:35:19.241268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233586    4614 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233589    4614 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233592    4614 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233595    4614 flags.go:64] FLAG: --pod-cidr=""
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233598    4614 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233601    4614 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233604    4614 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233606    4614 flags.go:64] FLAG: --port="10250"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233609    4614 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233612    4614 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-07c983f20e53ce506"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233615    4614 flags.go:64] FLAG: --qos-reserved=""
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233618    4614 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233621    4614 flags.go:64] FLAG: --register-node="true"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233624    4614 flags.go:64] FLAG: --register-with-taints=""
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233628    4614 flags.go:64] FLAG: --registry-burst="10"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233631    4614 flags.go:64] FLAG: --registry-qps="5"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233633    4614 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233636    4614 flags.go:64] FLAG: --reserved-memory=""
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233640    4614 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233644    4614 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233647    4614 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233650    4614 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233653    4614 flags.go:64] FLAG: --runonce="false"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233664    4614 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233668    4614 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233671    4614 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 09:35:19.241912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233674    4614 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233677    4614 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233680    4614 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233685    4614 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233689    4614 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233692    4614 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233695    4614 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233698    4614 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233700    4614 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233704    4614 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233707    4614 flags.go:64] FLAG: --system-cgroups=""
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233709    4614 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233714    4614 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233717    4614 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233721    4614 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233726    4614 flags.go:64] FLAG: --tls-min-version=""
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233728    4614 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233731    4614 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233734    4614 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233737    4614 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233740    4614 flags.go:64] FLAG: --v="2"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233744    4614 flags.go:64] FLAG: --version="false"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233748    4614 flags.go:64] FLAG: --vmodule=""
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233752    4614 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.233755    4614 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 09:35:19.242560 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233939    4614 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233943    4614 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233947    4614 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233950    4614 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233953    4614 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233956    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233959    4614 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233962    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233964    4614 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233967    4614 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233972    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233975    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233977    4614 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233980    4614 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233983    4614 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233986    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233988    4614 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233991    4614 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233993    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:19.243166 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233996    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.233999    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234001    4614 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234004    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234006    4614 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234009    4614 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234011    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234014    4614 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234017    4614 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234019    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234022    4614 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234024    4614 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234027    4614 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234031    4614 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234034    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234036    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234039    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234041    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234044    4614 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:19.243942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234046    4614 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234049    4614 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234051    4614 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234054    4614 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234058    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234067    4614 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234070    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234073    4614 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234076    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234078    4614 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234081    4614 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234084    4614 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234093    4614 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234097    4614 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234099    4614 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234102    4614 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234104    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234107    4614 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234109    4614 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234112    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:19.244511 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234116    4614 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234119    4614 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234123    4614 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234125    4614 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234128    4614 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234131    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234135    4614 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234137    4614 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234140    4614 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234143    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234145    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234148    4614 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234150    4614 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234153    4614 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234155    4614 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234158    4614 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234162    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234164    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234173    4614 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:19.245344 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234176    4614 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234178    4614 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234181    4614 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234183    4614 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234186    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234189    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234191    4614 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234194    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234196    4614 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234199    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234201    4614 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234204    4614 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234206    4614 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234209    4614 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234211    4614 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234214    4614 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234217    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234219    4614 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234222    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234228    4614 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:19.246113 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234231    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234233    4614 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234236    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234238    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234241    4614 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234243    4614 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234246    4614 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234249    4614 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234252    4614 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.234281    4614 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234330    4614 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234335    4614 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234343    4614 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234347    4614 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234361    4614 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234364    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:19.246674 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234367    4614 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234369    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234372    4614 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234375    4614 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234378    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234381    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234383    4614 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234386    4614 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234389    4614 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234392    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234394    4614 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234397    4614 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234399    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234402    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234405    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234407    4614 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234411    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234414    4614 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234416    4614 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:19.247340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234419    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234422    4614 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234424    4614 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234427    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234430    4614 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234432    4614 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234436    4614 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234438    4614 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234441    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234443    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234454    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234457    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234460    4614 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234462    4614 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234465    4614 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234467    4614 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234470    4614 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234472    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234475    4614 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:19.248195 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234478    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234480    4614 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234483    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234485    4614 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234488    4614 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234490    4614 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234493    4614 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234495    4614 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234498    4614 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234500    4614 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234505    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234508    4614 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234511    4614 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234513    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234518    4614 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234522    4614 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234525    4614 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234527    4614 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234530    4614 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:19.248911 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234533    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234535    4614 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234538    4614 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234540    4614 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234543    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234551    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234554    4614 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234557    4614 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234559    4614 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234562    4614 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234565    4614 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234567    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234570    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234572    4614 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234575    4614 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234577    4614 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234580    4614 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234582    4614 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234585    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:19.249423 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234587    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234590    4614 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234592    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234595    4614 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234599    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234601    4614 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234604    4614 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234606    4614 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234609    4614 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234611    4614 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234614    4614 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234616    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234619    4614 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234622    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234624    4614 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234627    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234629    4614 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234632    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234634    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234644    4614 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:19.249926 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234648    4614 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234650    4614 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234653    4614 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.234657    4614 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.239276    4614 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.239294    4614 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.239312    4614 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.239317    4614 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239392    4614 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239398    4614 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239401    4614 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239404    4614 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239407    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239409    4614 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239413    4614 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239415    4614 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239418    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239421    4614 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239423    4614 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239426    4614 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:19.250530 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239428    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239431    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239434    4614 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239437    4614 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239440    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239442    4614 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239446    4614 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239449    4614 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239452    4614 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239455    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239457    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239459    4614 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239462    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239465    4614 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239467    4614 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239470    4614 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239473    4614 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239477    4614 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239480    4614 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:19.251006 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239483    4614 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239485    4614 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239488    4614 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239491    4614 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239494    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239496    4614 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239499    4614 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239501    4614 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239504    4614 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239508    4614 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239512    4614 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239515    4614 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239518    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239521    4614 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239524    4614 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239527    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239531    4614 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239534    4614 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239536    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:19.251487 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239539    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239542    4614 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239544    4614 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239547    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239549    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239552    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239555    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239557    4614 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239560    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239562    4614 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239565    4614 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239568    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239571    4614 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239573    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239576    4614 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239578    4614 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239581    4614 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239585    4614 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239590    4614 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:19.251945 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239593    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239595    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239598    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239601    4614 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239603    4614 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239606    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239609    4614 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239612    4614 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239615    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239617    4614 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239620    4614 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239623    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239625    4614 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239628    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239631    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239633    4614 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239636    4614 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239638    4614 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239641    4614 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239643    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:19.252453 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239646    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239648    4614 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239651    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239653    4614 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239656    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239658    4614 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239661    4614 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239664    4614 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239667    4614 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239669    4614 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239672    4614 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239674    4614 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239677    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239679    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239682    4614 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239684    4614 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:19.252942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239687    4614 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.239714    4614 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239875    4614 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239879    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239883    4614 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239886    4614 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239889    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239892    4614 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239894    4614 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239896    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239899    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239901    4614 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239904    4614 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239907    4614 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239909    4614 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239912    4614 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:19.253326 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239914    4614 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239917    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239920    4614 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239922    4614 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239925    4614 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239927    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239930    4614 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239933    4614 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239936    4614 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239938    4614 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239940    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239943    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239946    4614 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239949    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239952    4614 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239955    4614 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239957    4614 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239960    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239962    4614 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239965    4614 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239968    4614 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:19.253745 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239970    4614 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239974    4614 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239978    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239981    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239983    4614 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239986    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239988    4614 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239991    4614 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239994    4614 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239996    4614 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.239998    4614 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240001    4614 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240004    4614 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240006    4614 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240009    4614 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240011    4614 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240014    4614 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240016    4614 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240019    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:19.254243 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240023    4614 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240027    4614 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240029    4614 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240032    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240035    4614 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240037    4614 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240040    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240043    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240045    4614 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240048    4614 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240050    4614 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240053    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240056    4614 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240058    4614 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240061    4614 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240064    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240066    4614 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240069    4614 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240071    4614 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:19.254712 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240074    4614 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240077    4614 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240079    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240082    4614 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240085    4614 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240087    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240090    4614 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240092    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240095    4614 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240097    4614 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240101    4614 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240103    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240106    4614 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240108    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240111    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240114    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240116    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240119    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240122    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:19.255169 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240125    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240127    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240130    4614 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240132    4614 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240135    4614 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240137    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240140    4614 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240142    4614 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240145    4614 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240147    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240150    4614 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240152    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240154    4614 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240157    4614 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.240179    4614 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 09:35:19.255704 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240209    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240214    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240217    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240220    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240223    4614 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240225    4614 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240228    4614 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240231    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240233    4614 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240235    4614 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240238    4614 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240240    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240243    4614 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240246    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240248    4614 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240251    4614 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240253    4614 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240256    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240259    4614 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 09:35:19.256131 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240262    4614 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240264    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240266    4614 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240269    4614 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240271    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240274    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240276    4614 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240279    4614 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240282    4614 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240284    4614 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240287    4614 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240289    4614 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240292    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240295    4614 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240297    4614 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240300    4614 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240302    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240305    4614 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240307    4614 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240310    4614 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 09:35:19.256663 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240312    4614 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240318    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240321    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240324    4614 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240326    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240329    4614 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240332    4614 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240335    4614 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240338    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240341    4614 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240343    4614 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240346    4614 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240361    4614 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240367    4614 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240371    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240374    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240377    4614 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240380    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240383    4614 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240385    4614 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 09:35:19.257147 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240388    4614 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240390    4614 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240393    4614 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240395    4614 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240398    4614 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240401    4614 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240404    4614 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240406    4614 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240409    4614 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240411    4614 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240414    4614 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240418    4614 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240421    4614 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240426    4614 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240430    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240432    4614 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240435    4614 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240438    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240441    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240443    4614 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 09:35:19.257693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240446    4614 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240449    4614 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240452    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240455    4614 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240457    4614 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240460    4614 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240462    4614 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240465    4614 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240468    4614 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240470    4614 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240473    4614 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240476    4614 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240478    4614 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240481    4614 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240483    4614 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240486    4614 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240488    4614 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240491    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240494    4614 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 09:35:19.258173 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240496    4614 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240499    4614 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240501    4614 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240504    4614 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240506    4614 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240510    4614 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240514    4614 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:19.240516    4614 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.240566    4614 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.241630    4614 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.241700    4614 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.241879    4614 server.go:1014] "Starting client certificate rotation"
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.242020    4614 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.242076    4614 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 09:29:37 +0000 UTC" deadline="2028-07-14 02:01:33.912384762 +0000 UTC"
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.242122    4614 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14872h26m14.670266832s"
Nov 02 09:35:19.258732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.242394    4614 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 09:35:19.259130 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.243391    4614 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 09:35:19.259130 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.246039    4614 log.go:25] "Validated CRI v1 runtime API"
Nov 02 09:35:19.259130 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.248385    4614 log.go:25] "Validated CRI v1 image API"
Nov 02 09:35:19.259130 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.249164    4614 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 09:35:19.259130 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.250964    4614 fs.go:135] Filesystem UUIDs: map[2024e37c-a86a-4ede-8331-96dbd215a633:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c503b405-9dea-45a0-86db-375ddf86232d:/dev/nvme0n1p3]
Nov 02 09:35:19.259272 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.250985    4614 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a1d871179fe9805f96aec4580350e40fb3e51ec6b03897a425ea92fc41195cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a1d871179fe9805f96aec4580350e40fb3e51ec6b03897a425ea92fc41195cf/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ae26e8b8c3e1c57f571c41870d9f6a47b5673f63567f8fd52730bc733d91f77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ae26e8b8c3e1c57f571c41870d9f6a47b5673f63567f8fd52730bc733d91f77/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24d082cb9734f62d614ce4fab957d985b65f9124bb800317b72a70da745147f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24d082cb9734f62d614ce4fab957d985b65f9124bb800317b72a70da745147f2/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25fadf7ae5035e6ea77cbd198cd7511ff7ac7c0db5a76eb943bab1dd1f8c7319/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25fadf7ae5035e6ea77cbd198cd7511ff7ac7c0db5a76eb943bab1dd1f8c7319/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f3f8a01c0a8c37fc568a03f5687d6ee6ab5d9f8dd8c0d5ff4ca053f30d7aeb5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f3f8a01c0a8c37fc568a03f5687d6ee6ab5d9f8dd8c0d5ff4ca053f30d7aeb5/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ff93a6f6cbd0c3e509d86d8ab2a2ada54b2a7eb471ac8c0aadef2d8a4a58961/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ff93a6f6cbd0c3e509d86d8ab2a2ada54b2a7eb471ac8c0aadef2d8a4a58961/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48a3b7e17d2c50cb55a8dac745b6af94c7bd3fa462c11fda0d38cb0e30daad72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48a3b7e17d2c50cb55a8dac745b6af94c7bd3fa462c11fda0d38cb0e30daad72/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b5684e60afbf8fd4dbca78f242bb6968081d37cea6f1a5529aa2c9fca61202f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b5684e60afbf8fd4dbca78f242bb6968081d37cea6f1a5529aa2c9fca61202f/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5378d6236e7ea1b301c2da5f0cd98cc06abd991f2cb580cba7d08b14abc5d3cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5378d6236e7ea1b301c2da5f0cd98cc06abd991f2cb580cba7d08b14abc5d3cc/userdata/shm major:0 minor:54 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c0719828efa2d70fa740192c6b713ae871a0172206cc80593be9364ae330fba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c0719828efa2d70fa740192c6b713ae871a0172206cc80593be9364ae330fba/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/878c16cc78814b967d23734afb4160c22728b32027f4a65d65d24aca63ebcd6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/878c16cc78814b967d23734afb4160c22728b32027f4a65d65d24aca63ebcd6b/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8aaa054d2f0b75d62b2740015291ba11af78a7a408113bc290610ad1326bf809/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8aaa054d2f0b75d62b2740015291ba11af78a7a408113bc290610ad1326bf809/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9392de63d13df86c30d47c92fc00921e42804aead1a6d0bdd53bba96c41ecd1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9392de63d13df86c30d47c92fc00921e42804aead1a6d0bdd53bba96c41ecd1c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94715adf5b967fbb49956da50a9842f338aa82b3bb78e71e7340064ca9010e3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94715adf5b967fbb49956da50a9842f338aa82b3bb78e71e7340064ca9010e3c/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94ae0eb45814cb611198f1782e1a3ff3853868f6846a90ee67c74e4dfdcdffe4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94ae0eb45814cb611198f1782e1a3ff3853868f6846a90ee67c74e4dfdcdffe4/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae2122bcae7d253da6f1dbaca0bd3c1a99f8f8fe90d1da4ab674d862b67ae07e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae2122bcae7d253da6f1dbaca0bd3c1a99f8f8fe90d1da4ab674d862b67ae07e/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb423a8e648d75df0e1c33716b4e847e2a8c486e2b4214692de72b015b184944/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb423a8e648d75df0e1c33716b4e847e2a8c486e2b4214692de72b015b184944/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdc069d72a1211d8c85fd25a5400db32e9412206edcdb9eb5f93661cc6fba78b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdc069d72a1211d8c85fd25a5400db32e9412206edcdb9eb5f93661cc6fba78b/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db15077c4cbec03878e701d7f36ac7f5f92af8065f0fbe7c233b74fc26da46b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db15077c4cbec03878e701d7f36ac7f5f92af8065f0fbe7c233b74fc26da46b9/userdata/shm major:0 minor:71 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e35a99be9186f5ac6d00d0ae645a12ab8c5a3aa4dafcf784eaadb76c61aa057c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e35a99be9186f5ac6d00d0ae645a12ab8c5a3aa4dafcf784eaadb76c61aa057c/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e53194c8e8afd2068f00b559903228fed6cb73e0d4677b13dac8803b9f0fe2c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e53194c8e8afd2068f00b559903228fed6cb73e0d4677b13dac8803b9f0fe2c3/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5451ba8c922662da7e07f82608496e5dad034111e57153ecbe5c8bce1d22f64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5451ba8c922662da7e07f82608496e5dad034111e57153ecbe5c8bce1d22f64/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09b69cf9-00f2-48f9-9c66-f22027a2376c/volumes/kubernetes.io~projected/kube-api-access-xz5dm:{mountpoint:/var/lib/kubelet/pods/09b69cf9-00f2-48f9-9c66-f22027a2376c/volumes/kubernetes.io~projected/kube-api-access-xz5dm major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/118b62a2-be2c-4119-87d4-914839ecc25a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/118b62a2-be2c-4119-87d4-914839ecc25a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/118b62a2-be2c-4119-87d4-914839ecc25a/volumes/kubernetes.io~projected/kube-api-access-xwtd7:{mountpoint:/var/lib/kubelet/pods/118b62a2-be2c-4119-87d4-914839ecc25a/volumes/kubernetes.io~projected/kube-api-access-xwtd7 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/118b62a2-be2c-4119-87d4-914839ecc25a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/118b62a2-be2c-4119-87d4-914839ecc25a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26adcc96-ef0b-416f-8b15-e01be59fb5ae/volumes/kubernetes.io~projected/kube-api-access-zr6kp:{mountpoint:/var/lib/kubelet/pods/26adcc96-ef0b-416f-8b15-e01be59fb5ae/volumes/kubernetes.io~projected/kube-api-access-zr6kp major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26adcc96-ef0b-416f-8b15-e01be59fb5ae/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/26adcc96-ef0b-416f-8b15-e01be59fb5ae/volumes/kubernetes.io~secret/cert major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d28887e-4aa9-4209-a51a-fec831136660/volumes/kubernetes.io~projected/kube-api-access-n8ms9:{mountpoint:/var/lib/kubelet/pods/2d28887e-4aa9-4209-a51a-fec831136660/volumes/kubernetes.io~projected/kube-api-access-n8ms9 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d28887e-4aa9-4209-a51a-fec831136660/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/2d28887e-4aa9-4209-a51a-fec831136660/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2daf8a59-8027-42b4-8414-55a0883bc04f/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/2daf8a59-8027-42b4-8414-55a0883bc04f/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2daf8a59-8027-42b4-8414-55a0883bc04f/volumes/kubernetes.io~projected/kube-api-access-v5ql9:{mountpoint:/var/lib/kubelet/pods/2daf8a59-8027-42b4-8414-55a0883bc04f/volumes/kubernetes.io~projected/kube-api-access-v5ql9 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2daf8a59-8027-42b4-8414-55a0883bc04f/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/2daf8a59-8027-42b4-8414-55a0883bc04f/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2daf8a59-8027-42b4-8414-55a0883bc04f/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/2daf8a59-8027-42b4-8414-55a0883bc04f/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3603dee5-a95f-47e4-9d60-381e82b8afa4/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3603dee5-a95f-47e4-9d60-381e82b8afa4/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3603dee5-a95f-47e4-9d60-381e82b8afa4/volumes/kubernetes.io~projected/kube-api-access-xhsmx:{mountpoint:/var/lib/kubelet/pods/3603dee5-a95f-47e4-9d60-381e82b8afa4/volumes/kubernetes.io~projected/kube-api-access-xhsmx major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3603dee5-a95f-47e4-9d60-381e82b8afa4/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3603dee5-a95f-47e4-9d60-381e82b8afa4/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3603dee5-a95f-47e4-9d60-381e82b8afa4/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3603dee5-a95f-47e4-9d60-381e82b8afa4/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c01c082-8479-460d-a80d-5e590ab2fcbe/volumes/kubernetes.io~projected/kube-api-access-7b2xg:{mountpoint:/var/lib/kubelet/pods/3c01c082-8479-460d-a80d-5e590ab2fcbe/volumes/kubernetes.io~projected/kube-api-access-7b2xg major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48aad7e1-623f-4eab-8a57-a50fefe7f2b1/volumes/kubernetes.io~projected/kube-api-access-cfmg8:{mountpoint:/var/lib/kubelet/pods/48aad7e1-623f-4eab-8a57-a50fefe7f2b1/volumes/kubernetes.io~projected/kube-api-access-cfmg8 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/502f9765-e116-4359-b513-c1d00c90d7de/volumes/kubernetes.io~projected/kube-api-access-jgmvv:{mountpoint:/var/lib/kubelet/pods/502f9765-e116-4359-b513-c1d00c90d7de/volumes/kubernetes.io~projected/kube-api-access-jgmvv major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8/volumes/kubernetes.io~projected/kube-api-access-zsbgj:{mountpoint:/var/lib/kubelet/pods/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8/volumes/kubernetes.io~projected/kube-api-access-zsbgj major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77530df7-545e-42b0-ad16-ce934cb10dab/volumes/kubernetes.io~projected/kube-api-access-pqvqm:{mountpoint:/var/lib/kubelet/pods/77530df7-545e-42b0-ad16-ce934cb10dab/volumes/kubernetes.io~projected/kube-api-access-pqvqm major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78cde7e0-a2f3-4f33-ba88-6c9fedc723eb/volumes/kubernetes.io~projected/kube-api-access-psmww:{mountpoint:/var/lib/kubelet/pods/78cde7e0-a2f3-4f33-ba88-6c9fedc723eb/volumes/kubernetes.io~projected/kube-api-access-psmww major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80746de3-f9f3-4efe-a963-9b37f290fcb7/volumes/kubernetes.io~projected/kube-api-access-q4zq4:{mountpoint:/var/lib/kubelet/pods/80746de3-f9f3-4efe-a963-9b37f290fcb7/volumes/kubernetes.io~projected/kube-api-access-q4zq4 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80746de3-f9f3-4efe-a963-9b37f290fcb7/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/80746de3-f9f3-4efe-a963-9b37f290fcb7/volumes/kubernetes.io~secret/default-certificate major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80746de3-f9f3-4efe-a963-9b37f290fcb7/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/80746de3-f9f3-4efe-a963-9b37f290fcb7/volumes/kubernetes.io~secret/stats-auth major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85741a00-b702-46cb-a90c-e000bf9976f0/volumes/kubernetes.io~projected/kube-api-access-b25xs:{mountpoint:/var/lib/kubelet/pods/85741a00-b702-46cb-a90c-e000bf9976f0/volumes/kubernetes.io~projected/kube-api-access-b25xs major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85741a00-b702-46cb-a90c-e000bf9976f0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/85741a00-b702-46cb-a90c-e000bf9976f0/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92f5b78b-58a0-4d0d-9bf6-97c4943597fd/volumes/kubernetes.io~projected/kube-api-access-bbxf5:{mountpoint:/var/lib/kubelet/pods/92f5b78b-58a0-4d0d-9bf6-97c4943597fd/volumes/kubernetes.io~projected/kube-api-access-bbxf5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95d7a9df-37e3-492b-9540-3df603931723/volumes/kubernetes.io~projected/kube-api-access-qvdcd:{mountpoint:/var/lib/kubelet/pods/95d7a9df-37e3-492b-9540-3df603931723/volumes/kubernetes.io~projected/kube-api-access-qvdcd major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/961f499a-2ed3-4e0d-b5d2-9194d18b696a/volumes/kubernetes.io~projected/kube-api-access-2zv58:{mountpoint:/var/lib/kubelet/pods/961f499a-2ed3-4e0d-b5d2-9194d18b696a/volumes/kubernetes.io~projected/kube-api-access-2zv58 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a05b1aec-cdf3-424b-8dfe-d4a6f06d6536/volumes/kubernetes.io~projected/kube-api-access-8htqg:{mountpoint:/var/lib/kubelet/pods/a05b1aec-cdf3-424b-8dfe-d4a6f06d6536/volumes/kubernetes.io~projected/kube-api-access-8htqg major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2608084-c9df-4998-a521-fb2239b3c349/volumes/kubernetes.io~projected/kube-api-access-zzrxx:{mountpoint:/var/lib/kubelet/pods/a2608084-c9df-4998-a521-fb2239b3c349/volumes/kubernetes.io~projected/kube-api-access-zzrxx major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2608084-c9df-4998-a521-fb2239b3c349/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a2608084-c9df-4998-a521-fb2239b3c349/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adfc6ef2-c0db-4fa7-9d91-95df203dff49/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/adfc6ef2-c0db-4fa7-9d91-95df203dff49/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adfc6ef2-c0db-4fa7-9d91-95df203dff49/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/adfc6ef2-c0db-4fa7-9d91-95df203dff49/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adfc6ef2-c0db-4fa7-9d91-95df203dff49/volumes/kubernetes.io~projected/kube-api-access-7wmsb:{mountpoint:/var/lib/kubelet/pods/adfc6ef2-c0db-4fa7-9d91-95df203dff49/volumes/kubernetes.io~projected/kube-api-access-7wmsb major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b36bfac6-a600-4029-a446-7de82575fb0a/volumes/kubernetes.io~projected/kube-api-access-n9tb2:{mountpoint:/var/lib/kubelet/pods/b36bfac6-a600-4029-a446-7de82575fb0a/volumes/kubernetes.io~projected/kube-api-access-n9tb2 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9ed0d41-1122-4ec2-a9af-e84cc71605df/volumes/kubernetes.io~projected/kube-api-access-pmw7c:{mountpoint:/var/lib/kubelet/pods/b9ed0d41-1122-4ec2-a9af-e84cc71605df/volumes/kubernetes.io~projected/kube-api-access-pmw7c major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5b95e7b-2ce0-45a1-9859-0d2378576064/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c5b95e7b-2ce0-45a1-9859-0d2378576064/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9ecca6d-3d39-4f10-9129-06feb4b292fa/volumes/kubernetes.io~projected/kube-api-access-nfscl:{mountpoint:/var/lib/kubelet/pods/d9ecca6d-3d39-4f10-9129-06feb4b292fa/volumes/kubernetes.io~projected/kube-api-access-nfscl major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbd4d998-a5be-4a0a-9113-8f3782df753a/volumes/kubernetes.io~projected/kube-api-access-b86xb:{mountpoint:/var/lib/kubelet/pods/dbd4d998-a5be-4a0a-9113-8f3782df753a/volumes/kubernetes.io~projected/kube-api-access-b86xb major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbd4d998-a5be-4a0a-9113-8f3782df753a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/dbd4d998-a5be-4a0a-9113-8f3782df753a/volumes/kubernetes.io~secret/serving-cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e/volumes/kubernetes.io~projected/kube-api-access-jfq4v:{mountpoint:/var/lib/kubelet/pods/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e/volumes/kubernetes.io~projected/kube-api-access-jfq4v major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8df50bb-8d26-42c1-b496-c9d9f56785d9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e8df50bb-8d26-42c1-b496-c9d9f56785d9/volumes/kubernetes.io~secret/agent-certs major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb/volumes/kubernetes.io~projected/kube-api-access-l6qw7:{mountpoint:/var/lib/kubelet/pods/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb/volumes/kubernetes.io~projected/kube-api-access-l6qw7 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fce2663d-12a8-427f-b8ec-0d2238a06470/volumes/kubernetes.io~projected/kube-api-access-qh7fg:{mountpoint:/var/lib/kubelet/pods/fce2663d-12a8-427f-b8ec-0d2238a06470/volumes/kubernetes.io~projected/kube-api-access-qh7fg major:0 minor:205 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ff70cd6ea71a578430d058c8aeedbba730939b6641f3e3f598fa009c119ff7dc/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8e32e5a1cb17e4020d7a978a93e23af36e84c4107785e6a36e1b9c898c8ba4da/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b0b03a5a485ba0069a188629446f16c8a3c9e76bc9594f18ebd97e0b3515b616/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/2b371ab9f60865cf238c005df245b954795612ba7aa4456e095da220d3131d89/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/87d348e89dfdf3d3e5292cd386e3dd84f65f895f6746bd7ebaf71e41138cb1a0/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/59f24971a2097916aa9cd884eca6960b5c3381dc73b9d5e1ccfc27ae2a53a65c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/62452c13c14f45217e18447d32221419ccdf328275a41ddc04d953909713f7d7/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3c27fd1597aace8cef6b581952267fdde781b11011128c7b5f69df67eb872557/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f36ae650a1f55655d5852e24a206bce799c1a2add687843cbad03f4a55486b72/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/e31313bd6cd76449895d728bf00ce8146eb0274b8949fa5a9de3b8e3a8528832/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/f577ac7f05745b5d08584ea430f09426f0000feab8fce53c4d20b9f5abe02d46/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/35733788496af208bdc455ec21b17078c7c53c58523f278f50900f9b1443b2f7/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/bdb7ec5081c0c0eb5c88736c0a7f531bd23965db611151e12d59c2e1ec0b8628/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/2e0ef2d03c7c0b0e60e1f56e566969ee04eb80fe2e4ba74166d65449debf8078/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/607d1059687ab44e864aa0f27456a8c79c14e8bf42a8226747d34e8896ec5daa/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/6b3ad4256aa2c8fefe3e537b4f69e72bd8961fc55b92b238a31953cc4522694f/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/873462e32ca931ea562ba414b5245a01cc5fc5ea9394a0b026a1f29630551843/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/2dd7923c313631d8af3aa6b4e03966776f0ffd8a9cccd1af47ab6d2525e17b5e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/218206df1b6450dbc598172d10565706bce9b85f1bbef415d2bd605cc9ffa7eb/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c8480cd5f303e95c39322105ef93ef00aa70bb6cd1f5c086a177568e2f684459/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e5fb7cbb704fdb0c0fc0c19d97b9ff7186b209cb24f738171c6b812951a5553d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/144c410835ce5a6e1912648cae197a54156b0f63c3bb4e826781a410637d46af/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/ee01fa67206a886108393bce0e61a573fba45961e207ff2f1518ffaaac67af25/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/dea6eafa8a0fcfd526168427151f661de2eb917523aac8561dce0d487d9ab0e8/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/ab7c8fdcf8d28603724ef5b58a67ce868341fa3fe8d532910348785d86e1cc22/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/f9f778a295d13a713a3749771964707a5a099d19ea696f53fbff3b9d6f072995/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/3fcb6a0420c155f05f3185eb46c78588c458796df57b8407221ee9002685a3f1/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/1e2015018920d5f3c904b766691e313067936ba89d6c3cdee582feb6a90bb0d7/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/173faa15b686628a419de28671070c60c5930ee5505703ec94cfedee9dfa0822/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/cd0103b0082c3624ec3217785696c154d2939f33bf10adab2e1bca25fb41ff2d/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/6c57d8368263551b981dbf5e339e96fefa90c88cdbdf99d663e4f088b166eb5e/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/1e654b369381ee42d742c80adaa2de685559d34dae8431e8e89fa54ef760eb5d/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/83655194f1cfa202698ce4f14574c27b313f6a9ca80e85c2aec8fbac4707c77f/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/cd1dd9300a7365e1934f0833f8f35ec6d7acfefcad0aedd57376d0665ec93de7/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/693f9b0ef0a1fc602b9eb50ef1e8216ae15fd9c2dd4a26b9cbf443c91130d235/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/fba59d282bb932902bed1dfedcecd08c2750ad0057e22c4c8393a5932f08e4be/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/a611888c36132f216e859edf0c62b27804d700adda777dbd88cdebc5cd95c1d7/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/c25b74cc719e78ea4a01c5152045fc94c0b33c58f8f02bb853655f3035d59588/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/d1d82317620ab17d00512b67fcd79ccb3ea3e0b3324261196d824d423201a7cd/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/b168eaa14b0c6a0f827c5637af7852e9f869e5491f0dbaafa09a01696cc7d786/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/ff6f3c98de16db16954b7f6da941d04e68c5672fae277a39460fa0698dee289a/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/61f172ac33b131db0cc7f4a7a795be47a0bcb03df03a2d7c4ada98099ba90493/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/213cfeeae3b7e6c2667cfdc3f703a223898fccfe2f516fc19b9c3bd3bb75e82d/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/58d241c4edef953fd8523b94f82c7e0d7fffe1ce800321f0cceb83853758211f/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/2d73c898d993a7693b4f335bda1cb3d87ddb66242fa94f6c75bd3eaab3125d71/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/a04192a988f3df742e4e4e39a0ee3027f783d6acdbeffc377d743b8a2ebc4b92/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/71a774216c623b4e5d41c45c6b1c33276fc32333ed446e11305654f28ade7b21/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-73:{mountpoint:/var/lib/containers/storage/overlay/18baf58a93ee2f3d615ab7fb5a6075b20a363bb032bc9e531fef5479774364cc/merged major:0 minor:73 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/26296a2b6585abf4050d4378f6e1480bd83001e9474d5f73d868fc4ddede0f5c/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/ff4f20c8d5943d7534760144c53347cfdb62784b219805dd82927ad9539b75b1/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/fbf4f249fff2e784dc6a04f5e075f614ab76c75b7be0fec5b1e82b8b13ca3f53/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/129d61873aa90dfe1f1c6f22828de0bf252b2cf775cbace4feef944abb8e8fc9/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/167393b1878f2932faafa1b635b29bb8348b29ad7240595b3ad210d1d8e4f3a4/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/65227643a565de0b034b453884dce79b707b2c1239630a7ccfb72ad97ec02890/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 09:35:19.259502 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.258538    4614 manager.go:217] Machine: {Timestamp:2026-11-02 09:35:19.258075997 +0000 UTC m=+0.090734068 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099801 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f158347127cc5ae4467fed90ef1ee SystemUUID:ec2f1583-4712-7cc5-ae44-67fed90ef1ee BootID:8da1a5d0-6c11-4a2e-a6d1-64b974c4e3ea Filesystems:[{Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2608084-c9df-4998-a521-fb2239b3c349/volumes/kubernetes.io~projected/kube-api-access-zzrxx DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94ae0eb45814cb611198f1782e1a3ff3853868f6846a90ee67c74e4dfdcdffe4/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b9ed0d41-1122-4ec2-a9af-e84cc71605df/volumes/kubernetes.io~projected/kube-api-access-pmw7c DeviceMajor:0 DeviceMinor:233 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f3f8a01c0a8c37fc568a03f5687d6ee6ab5d9f8dd8c0d5ff4ca053f30d7aeb5/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d28887e-4aa9-4209-a51a-fec831136660/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/85741a00-b702-46cb-a90c-e000bf9976f0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3603dee5-a95f-47e4-9d60-381e82b8afa4/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adfc6ef2-c0db-4fa7-9d91-95df203dff49/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db15077c4cbec03878e701d7f36ac7f5f92af8065f0fbe7c233b74fc26da46b9/userdata/shm DeviceMajor:0 DeviceMinor:71 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78cde7e0-a2f3-4f33-ba88-6c9fedc723eb/volumes/kubernetes.io~projected/kube-api-access-psmww DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e53194c8e8afd2068f00b559903228fed6cb73e0d4677b13dac8803b9f0fe2c3/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/502f9765-e116-4359-b513-c1d00c90d7de/volumes/kubernetes.io~projected/kube-api-access-jgmvv DeviceMajor:0 DeviceMinor:243 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/77530df7-545e-42b0-ad16-ce934cb10dab/volumes/kubernetes.io~projected/kube-api-access-pqvqm DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a1d871179fe9805f96aec4580350e40fb3e51ec6b03897a425ea92fc41195cf/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3603dee5-a95f-47e4-9d60-381e82b8afa4/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/2daf8a59-8027-42b4-8414-55a0883bc04f/volumes/kubernetes.io~projected/kube-api-access-v5ql9 DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dbd4d998-a5be-4a0a-9113-8f3782df753a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/118b62a2-be2c-4119-87d4-914839ecc25a/volumes/kubernetes.io~projected/kube-api-access-xwtd7 DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adfc6ef2-c0db-4fa7-9d91-95df203dff49/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c0719828efa2d70fa740192c6b713ae871a0172206cc80593be9364ae330fba/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-73 DeviceMajor:0 DeviceMinor:73 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adfc6ef2-c0db-4fa7-9d91-95df203dff49/volumes/kubernetes.io~projected/kube-api-access-7wmsb DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8aaa054d2f0b75d62b2740015291ba11af78a7a408113bc290610ad1326bf809/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48a3b7e17d2c50cb55a8dac745b6af94c7bd3fa462c11fda0d38cb0e30daad72/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/961f499a-2ed3-4e0d-b5d2-9194d18b696a/volumes/kubernetes.io~projected/kube-api-access-2zv58 DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbd4d998-a5be-4a0a-9113-8f3782df753a/volumes/kubernetes.io~projected/kube-api-access-b86xb DeviceMajor:0 DeviceMinor:234 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fce2663d-12a8-427f-b8ec-0d2238a06470/volumes/kubernetes.io~projected/kube-api-access-qh7fg DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2daf8a59-8027-42b4-8414-55a0883bc04f/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e35a99be9186f5ac6d00d0ae645a12ab8c5a3aa4dafcf784eaadb76c61aa057c/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/48aad7e1-623f-4eab-8a57-a50fefe7f2b1/volumes/kubernetes.io~projected/kube-api-access-cfmg8 DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a05b1aec-cdf3-424b-8dfe-d4a6f06d6536/volumes/kubernetes.io~projected/kube-api-access-8htqg DeviceMajor:0 DeviceMinor:232 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24d082cb9734f62d614ce4fab957d985b65f9124bb800317b72a70da745147f2/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e/volumes/kubernetes.io~projected/kube-api-access-jfq4v DeviceMajor:0 DeviceMinor:287 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/95d7a9df-37e3-492b-9540-3df603931723/volumes/kubernetes.io~projected/kube-api-access-qvdcd DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b36bfac6-a600-4029-a446-7de82575fb0a/volumes/kubernetes.io~projected/kube-api-access-n9tb2 DeviceMajor:0 DeviceMinor:310 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d9ecca6d-3d39-4f10-9129-06feb4b292fa/volumes/kubernetes.io~projected/kube-api-access-nfscl DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ff93a6f6cbd0c3e509d86d8ab2a2ada54b2a7eb471ac8c0aadef2d8a4a58961/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/118b62a2-be2c-4119-87d4-914839ecc25a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/80746de3-f9f3-4efe-a963-9b37f290fcb7/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c5b95e7b-2ce0-45a1-9859-0d2378576064/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/09b69cf9-00f2-48f9-9c66-f22027a2376c/volumes/kubernetes.io~projected/kube-api-access-xz5dm DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9392de63d13df86c30d47c92fc00921e42804aead1a6d0bdd53bba96c41ecd1c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8/volumes/kubernetes.io~projected/kube-api-access-zsbgj DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26adcc96-ef0b-416f-8b15-e01be59fb5ae/volumes/kubernetes.io~projected/kube-api-access-zr6kp DeviceMajor:0 DeviceMinor:231 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/878c16cc78814b967d23734afb4160c22728b32027f4a65d65d24aca63ebcd6b/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/92f5b78b-58a0-4d0d-9bf6-97c4943597fd/volumes/kubernetes.io~projected/kube-api-access-bbxf5 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d28887e-4aa9-4209-a51a-fec831136660/volumes/kubernetes.io~projected/kube-api-access-n8ms9 DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2daf8a59-8027-42b4-8414-55a0883bc04f/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae2122bcae7d253da6f1dbaca0bd3c1a99f8f8fe90d1da4ab674d862b67ae07e/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/26adcc96-ef0b-416f-8b15-e01be59fb5ae/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94715adf5b967fbb49956da50a9842f338aa82b3bb78e71e7340064ca9010e3c/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c01c082-8479-460d-a80d-5e590ab2fcbe/volumes/kubernetes.io~projected/kube-api-access-7b2xg DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80746de3-f9f3-4efe-a963-9b37f290fcb7/volumes/kubernetes.io~projected/kube-api-access-q4zq4 DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25fadf7ae5035e6ea77cbd198cd7511ff7ac7c0db5a76eb943bab1dd1f8c7319/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/118b62a2-be2c-4119-87d4-914839ecc25a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2608084-c9df-4998-a521-fb2239b3c349/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b5684e60afbf8fd4dbca78f242bb6968081d37cea6f1a5529aa2c9fca61202f/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2daf8a59-8027-42b4-8414-55a0883bc04f/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5451ba8c922662da7e07f82608496e5dad034111e57153ecbe5c8bce1d22f64/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3603dee5-a95f-47e4-9d60-381e82b8afa4/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb423a8e648d75df0e1c33716b4e847e2a8c486e2b4214692de72b015b184944/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3603dee5-a95f-47e4-9d60-381e82b8afa4/volumes/kubernetes.io~projected/kube-api-access-xhsmx DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8df50bb-8d26-42c1-b496-c9d9f56785d9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdc069d72a1211d8c85fd25a5400db32e9412206edcdb9eb5f93661cc6fba78b/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/85741a00-b702-46cb-a90c-e000bf9976f0/volumes/kubernetes.io~projected/kube-api-access-b25xs DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/80746de3-f9f3-4efe-a963-9b37f290fcb7/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5378d6236e7ea1b301c2da5f0cd98cc06abd991f2cb580cba7d08b14abc5d3cc/userdata/shm DeviceMajor:0 DeviceMinor:54 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ae26e8b8c3e1c57f571c41870d9f6a47b5673f63567f8fd52730bc733d91f77/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb/volumes/kubernetes.io~projected/kube-api-access-l6qw7 DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:24d082cb9734f62 MacAddress:b6:7e:45:1e:82:1a Speed:10000 Mtu:8901} {Name:25fadf7ae5035e6 MacAddress:32:d1:ee:1a:15:88 Speed:10000 Mtu:8901} {Name:2f3f8a01c0a8c37 MacAddress:02:c3:01:bd:af:b9 Speed:10000 Mtu:8901} {Name:48a3b7e17d2c50c MacAddress:ee:8c:8a:21:d5:38 Speed:10000 Mtu:8901} {Name:4b5684e60afbf8f MacAddress:8a:d0:43:5f:b6:35 Speed:10000 Mtu:8901} {Name:878c16cc78814b9 MacAddress:52:66:6e:d8:fd:92 Speed:10000 Mtu:8901} {Name:94ae0eb45814cb6 MacAddress:6a:6f:10:9c:bb:11 Speed:10000 Mtu:8901} {Name:ae2122bcae7d253 MacAddress:b2:a0:11:37:49:69 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d5:66:2a:01 Speed:0 Mtu:9001} {Name:br-int MacAddress:12:ec:23:86:e1:89 Speed:0 Mtu:8901} {Name:cb423a8e648d75d MacAddress:b2:1c:91:f2:d8:3d Speed:10000 Mtu:8901} {Name:e35a99be9186f5a MacAddress:9a:69:ea:de:27:2c Speed:10000 Mtu:8901} {Name:e53194c8e8afd20 MacAddress:1a:69:b8:bf:74:f8 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d5:66:2a:01 Speed:0 Mtu:9001} {Name:f5451ba8c922662 MacAddress:d6:01:19:39:b3:56 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ce:a4:3e:95:d1:e7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:aa:25:31:7c:af:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 09:35:19.259502 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.258968    4614 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 09:35:19.259502 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.259044    4614 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 09:35:19.259502 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.259382    4614 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 09:35:19.259749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.259403    4614 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-174.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 09:35:19.259749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.259528    4614 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 09:35:19.259749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.259536    4614 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 09:35:19.259749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.259555    4614 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 09:35:19.259749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.259573    4614 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 09:35:19.259749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.259643    4614 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 09:35:19.259918 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.259801    4614 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 09:35:19.260092 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.260080    4614 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 09:35:19.260124 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.260119    4614 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 09:35:19.260163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.260132    4614 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 09:35:19.260163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.260142    4614 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 09:35:19.260163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.260151    4614 kubelet.go:405] "Adding apiserver pod source"
Nov 02 09:35:19.260163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.260160    4614 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 09:35:19.260803 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.260788    4614 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 09:35:19.260803 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.260802    4614 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 09:35:19.261234 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.261219    4614 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 09:35:19.263847 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.263825    4614 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 09:35:19.263847 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.263847    4614 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 09:35:19.264002 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.263989    4614 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 09:35:19.264313 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264295    4614 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 09:35:19.264387 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264348    4614 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 09:35:19.264543 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264522    4614 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 09:35:19.264646 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264573    4614 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 09:35:19.264646 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264581    4614 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 09:35:19.264646 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264588    4614 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 09:35:19.264646 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264594    4614 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 09:35:19.264646 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264601    4614 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 09:35:19.264646 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264608    4614 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 09:35:19.264646 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264614    4614 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 09:35:19.264646 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264626    4614 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 09:35:19.264646 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264632    4614 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 09:35:19.264646 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264641    4614 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 09:35:19.264646 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264651    4614 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 09:35:19.264998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264679    4614 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 09:35:19.264998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.264686    4614 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 09:35:19.268622 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.268607    4614 server.go:1263] "Started kubelet"
Nov 02 09:35:19.268707 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.268675    4614 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 09:35:19.269162 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.269078    4614 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 09:35:19.269249 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.269182    4614 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 09:35:19.269474 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.269456    4614 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 09:35:19.269464 ip-10-0-134-174 systemd[1]: Started Kubernetes Kubelet.
Nov 02 09:35:19.269939 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.269778    4614 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 09:35:19.272478 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.272462    4614 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 09:35:19.272556 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.272479    4614 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 09:35:19.272767 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.272622    4614 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 09:29:38 +0000 UTC" deadline="2028-05-04 06:01:42.802599262 +0000 UTC"
Nov 02 09:35:19.272767 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.272670    4614 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13172h26m23.529932886s"
Nov 02 09:35:19.272767 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.272705    4614 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 09:35:19.272767 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.272715    4614 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 09:35:19.272767 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:19.272699    4614 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-174.us-east-2.compute.internal\" not found"
Nov 02 09:35:19.273001 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.272872    4614 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 09:35:19.274731 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.274712    4614 factory.go:55] Registering systemd factory
Nov 02 09:35:19.274824 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.274735    4614 factory.go:223] Registration of the systemd container factory successfully
Nov 02 09:35:19.275062 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.275047    4614 factory.go:153] Registering CRI-O factory
Nov 02 09:35:19.275062 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.275063    4614 factory.go:223] Registration of the crio container factory successfully
Nov 02 09:35:19.275177 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.275129    4614 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 09:35:19.275177 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.275153    4614 factory.go:103] Registering Raw factory
Nov 02 09:35:19.277616 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.277601    4614 manager.go:1199] Started watching for new ooms in manager
Nov 02 09:35:19.278123 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.278104    4614 manager.go:319] Starting recovery of all containers
Nov 02 09:35:19.281828 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.281788    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8df50bb-8d26-42c1-b496-c9d9f56785d9" volumeName="kubernetes.io/secret/e8df50bb-8d26-42c1-b496-c9d9f56785d9-agent-certs" seLinuxMountContext=""
Nov 02 09:35:19.281925 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.281837    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" volumeName="kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 09:35:19.281925 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.281854    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2608084-c9df-4998-a521-fb2239b3c349" volumeName="kubernetes.io/secret/a2608084-c9df-4998-a521-fb2239b3c349-serving-cert" seLinuxMountContext=""
Nov 02 09:35:19.281925 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.281870    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbd4d998-a5be-4a0a-9113-8f3782df753a" volumeName="kubernetes.io/projected/dbd4d998-a5be-4a0a-9113-8f3782df753a-kube-api-access-b86xb" seLinuxMountContext=""
Nov 02 09:35:19.281925 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.281888    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3603dee5-a95f-47e4-9d60-381e82b8afa4" volumeName="kubernetes.io/empty-dir/3603dee5-a95f-47e4-9d60-381e82b8afa4-ca-trust-extracted" seLinuxMountContext=""
Nov 02 09:35:19.281925 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.281900    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78cde7e0-a2f3-4f33-ba88-6c9fedc723eb" volumeName="kubernetes.io/projected/78cde7e0-a2f3-4f33-ba88-6c9fedc723eb-kube-api-access-psmww" seLinuxMountContext=""
Nov 02 09:35:19.282160 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.281917    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80746de3-f9f3-4efe-a963-9b37f290fcb7" volumeName="kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-default-certificate" seLinuxMountContext=""
Nov 02 09:35:19.282160 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.281966    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92f5b78b-58a0-4d0d-9bf6-97c4943597fd" volumeName="kubernetes.io/projected/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-kube-api-access-bbxf5" seLinuxMountContext=""
Nov 02 09:35:19.282160 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282000    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9ed0d41-1122-4ec2-a9af-e84cc71605df" volumeName="kubernetes.io/projected/b9ed0d41-1122-4ec2-a9af-e84cc71605df-kube-api-access-pmw7c" seLinuxMountContext=""
Nov 02 09:35:19.282160 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282017    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3603dee5-a95f-47e4-9d60-381e82b8afa4" volumeName="kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-certificates" seLinuxMountContext=""
Nov 02 09:35:19.282160 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282041    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3603dee5-a95f-47e4-9d60-381e82b8afa4" volumeName="kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-bound-sa-token" seLinuxMountContext=""
Nov 02 09:35:19.282160 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282057    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3603dee5-a95f-47e4-9d60-381e82b8afa4" volumeName="kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 09:35:19.282160 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282087    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2daf8a59-8027-42b4-8414-55a0883bc04f" volumeName="kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-bound-sa-token" seLinuxMountContext=""
Nov 02 09:35:19.282519 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282443    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3603dee5-a95f-47e4-9d60-381e82b8afa4" volumeName="kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-trusted-ca" seLinuxMountContext=""
Nov 02 09:35:19.282519 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282468    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e2f49f4-2884-4804-bb3a-3ee463eeb4f8" volumeName="kubernetes.io/configmap/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-config" seLinuxMountContext=""
Nov 02 09:35:19.282519 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282479    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b36bfac6-a600-4029-a446-7de82575fb0a" volumeName="kubernetes.io/empty-dir/b36bfac6-a600-4029-a446-7de82575fb0a-data-volume" seLinuxMountContext=""
Nov 02 09:35:19.282519 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282491    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5b95e7b-2ce0-45a1-9859-0d2378576064" volumeName="kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret" seLinuxMountContext=""
Nov 02 09:35:19.282519 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282500    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77530df7-545e-42b0-ad16-ce934cb10dab" volumeName="kubernetes.io/configmap/77530df7-545e-42b0-ad16-ce934cb10dab-serviceca" seLinuxMountContext=""
Nov 02 09:35:19.282519 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282510    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80746de3-f9f3-4efe-a963-9b37f290fcb7" volumeName="kubernetes.io/projected/80746de3-f9f3-4efe-a963-9b37f290fcb7-kube-api-access-q4zq4" seLinuxMountContext=""
Nov 02 09:35:19.282519 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282521    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbd4d998-a5be-4a0a-9113-8f3782df753a" volumeName="kubernetes.io/configmap/dbd4d998-a5be-4a0a-9113-8f3782df753a-config" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282530    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="118b62a2-be2c-4119-87d4-914839ecc25a" volumeName="kubernetes.io/projected/118b62a2-be2c-4119-87d4-914839ecc25a-kube-api-access-xwtd7" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282542    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26adcc96-ef0b-416f-8b15-e01be59fb5ae" volumeName="kubernetes.io/projected/26adcc96-ef0b-416f-8b15-e01be59fb5ae-kube-api-access-zr6kp" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282560    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adfc6ef2-c0db-4fa7-9d91-95df203dff49" volumeName="kubernetes.io/empty-dir/adfc6ef2-c0db-4fa7-9d91-95df203dff49-tmp" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282571    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09b69cf9-00f2-48f9-9c66-f22027a2376c" volumeName="kubernetes.io/configmap/09b69cf9-00f2-48f9-9c66-f22027a2376c-cni-binary-copy" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282580    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09b69cf9-00f2-48f9-9c66-f22027a2376c" volumeName="kubernetes.io/configmap/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-daemon-config" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282593    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26adcc96-ef0b-416f-8b15-e01be59fb5ae" volumeName="kubernetes.io/secret/26adcc96-ef0b-416f-8b15-e01be59fb5ae-cert" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282633    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c01c082-8479-460d-a80d-5e590ab2fcbe" volumeName="kubernetes.io/configmap/3c01c082-8479-460d-a80d-5e590ab2fcbe-telemetry-config" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282662    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e2f49f4-2884-4804-bb3a-3ee463eeb4f8" volumeName="kubernetes.io/projected/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-kube-api-access-zsbgj" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282694    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85741a00-b702-46cb-a90c-e000bf9976f0" volumeName="kubernetes.io/configmap/85741a00-b702-46cb-a90c-e000bf9976f0-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282710    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85741a00-b702-46cb-a90c-e000bf9976f0" volumeName="kubernetes.io/empty-dir/85741a00-b702-46cb-a90c-e000bf9976f0-tmp" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282722    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92f5b78b-58a0-4d0d-9bf6-97c4943597fd" volumeName="kubernetes.io/empty-dir/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-tmp-dir" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282740    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d28887e-4aa9-4209-a51a-fec831136660" volumeName="kubernetes.io/configmap/2d28887e-4aa9-4209-a51a-fec831136660-trusted-ca" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282754    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2daf8a59-8027-42b4-8414-55a0883bc04f" volumeName="kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-trusted-ca" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282773    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2daf8a59-8027-42b4-8414-55a0883bc04f" volumeName="kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282787    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2daf8a59-8027-42b4-8414-55a0883bc04f" volumeName="kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-installation-pull-secrets" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282800    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85741a00-b702-46cb-a90c-e000bf9976f0" volumeName="kubernetes.io/empty-dir/85741a00-b702-46cb-a90c-e000bf9976f0-snapshots" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282821    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbd4d998-a5be-4a0a-9113-8f3782df753a" volumeName="kubernetes.io/secret/dbd4d998-a5be-4a0a-9113-8f3782df753a-serving-cert" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282835    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" volumeName="kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 09:35:19.282850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282853    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="118b62a2-be2c-4119-87d4-914839ecc25a" volumeName="kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282867    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d28887e-4aa9-4209-a51a-fec831136660" volumeName="kubernetes.io/projected/2d28887e-4aa9-4209-a51a-fec831136660-kube-api-access-n8ms9" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282886    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77530df7-545e-42b0-ad16-ce934cb10dab" volumeName="kubernetes.io/projected/77530df7-545e-42b0-ad16-ce934cb10dab-kube-api-access-pqvqm" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282911    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2daf8a59-8027-42b4-8414-55a0883bc04f" volumeName="kubernetes.io/empty-dir/2daf8a59-8027-42b4-8414-55a0883bc04f-ca-trust-extracted" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282924    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adfc6ef2-c0db-4fa7-9d91-95df203dff49" volumeName="kubernetes.io/projected/adfc6ef2-c0db-4fa7-9d91-95df203dff49-kube-api-access-7wmsb" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282942    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8df50bb-8d26-42c1-b496-c9d9f56785d9" volumeName="kubernetes.io/configmap/e8df50bb-8d26-42c1-b496-c9d9f56785d9-konnectivity-ca" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282957    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09b69cf9-00f2-48f9-9c66-f22027a2376c" volumeName="kubernetes.io/projected/09b69cf9-00f2-48f9-9c66-f22027a2376c-kube-api-access-xz5dm" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282974    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="118b62a2-be2c-4119-87d4-914839ecc25a" volumeName="kubernetes.io/secret/118b62a2-be2c-4119-87d4-914839ecc25a-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.282988    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d28887e-4aa9-4209-a51a-fec831136660" volumeName="kubernetes.io/configmap/2d28887e-4aa9-4209-a51a-fec831136660-config" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283000    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2daf8a59-8027-42b4-8414-55a0883bc04f" volumeName="kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-kube-api-access-v5ql9" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283017    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="502f9765-e116-4359-b513-c1d00c90d7de" volumeName="kubernetes.io/configmap/502f9765-e116-4359-b513-c1d00c90d7de-config-volume" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283029    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="502f9765-e116-4359-b513-c1d00c90d7de" volumeName="kubernetes.io/projected/502f9765-e116-4359-b513-c1d00c90d7de-kube-api-access-jgmvv" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283047    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80746de3-f9f3-4efe-a963-9b37f290fcb7" volumeName="kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-stats-auth" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283058    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85741a00-b702-46cb-a90c-e000bf9976f0" volumeName="kubernetes.io/projected/85741a00-b702-46cb-a90c-e000bf9976f0-kube-api-access-b25xs" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283071    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3603dee5-a95f-47e4-9d60-381e82b8afa4" volumeName="kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-installation-pull-secrets" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283380    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d28887e-4aa9-4209-a51a-fec831136660" volumeName="kubernetes.io/secret/2d28887e-4aa9-4209-a51a-fec831136660-serving-cert" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283397    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="961f499a-2ed3-4e0d-b5d2-9194d18b696a" volumeName="kubernetes.io/projected/961f499a-2ed3-4e0d-b5d2-9194d18b696a-kube-api-access-2zv58" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283412    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2608084-c9df-4998-a521-fb2239b3c349" volumeName="kubernetes.io/configmap/a2608084-c9df-4998-a521-fb2239b3c349-config" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283437    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b36bfac6-a600-4029-a446-7de82575fb0a" volumeName="kubernetes.io/projected/b36bfac6-a600-4029-a446-7de82575fb0a-kube-api-access-n9tb2" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283448    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9ecca6d-3d39-4f10-9129-06feb4b292fa" volumeName="kubernetes.io/configmap/d9ecca6d-3d39-4f10-9129-06feb4b292fa-iptables-alerter-script" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283458    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e" volumeName="kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283467    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5393080-e20a-47fc-8dc7-125f42457736" volumeName="kubernetes.io/configmap/e5393080-e20a-47fc-8dc7-125f42457736-nginx-conf" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283478    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2daf8a59-8027-42b4-8414-55a0883bc04f" volumeName="kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-certificates" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283486    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3603dee5-a95f-47e4-9d60-381e82b8afa4" volumeName="kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-kube-api-access-xhsmx" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283497    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85741a00-b702-46cb-a90c-e000bf9976f0" volumeName="kubernetes.io/secret/85741a00-b702-46cb-a90c-e000bf9976f0-serving-cert" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283505    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9ecca6d-3d39-4f10-9129-06feb4b292fa" volumeName="kubernetes.io/projected/d9ecca6d-3d39-4f10-9129-06feb4b292fa-kube-api-access-nfscl" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283514    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" volumeName="kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cni-binary-copy" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283539    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="118b62a2-be2c-4119-87d4-914839ecc25a" volumeName="kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-env-overrides" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283548    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c01c082-8479-460d-a80d-5e590ab2fcbe" volumeName="kubernetes.io/projected/3c01c082-8479-460d-a80d-5e590ab2fcbe-kube-api-access-7b2xg" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283560    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48aad7e1-623f-4eab-8a57-a50fefe7f2b1" volumeName="kubernetes.io/projected/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-kube-api-access-cfmg8" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283569    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95d7a9df-37e3-492b-9540-3df603931723" volumeName="kubernetes.io/projected/95d7a9df-37e3-492b-9540-3df603931723-kube-api-access-qvdcd" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283577    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2608084-c9df-4998-a521-fb2239b3c349" volumeName="kubernetes.io/projected/a2608084-c9df-4998-a521-fb2239b3c349-kube-api-access-zzrxx" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283589    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" volumeName="kubernetes.io/projected/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-kube-api-access-l6qw7" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283599    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fce2663d-12a8-427f-b8ec-0d2238a06470" volumeName="kubernetes.io/projected/fce2663d-12a8-427f-b8ec-0d2238a06470-kube-api-access-qh7fg" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283610    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="502f9765-e116-4359-b513-c1d00c90d7de" volumeName="kubernetes.io/empty-dir/502f9765-e116-4359-b513-c1d00c90d7de-tmp-dir" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283619    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85741a00-b702-46cb-a90c-e000bf9976f0" volumeName="kubernetes.io/configmap/85741a00-b702-46cb-a90c-e000bf9976f0-service-ca-bundle" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283628    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a05b1aec-cdf3-424b-8dfe-d4a6f06d6536" volumeName="kubernetes.io/projected/a05b1aec-cdf3-424b-8dfe-d4a6f06d6536-kube-api-access-8htqg" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283639    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="118b62a2-be2c-4119-87d4-914839ecc25a" volumeName="kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-ovnkube-config" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283654    4614 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adfc6ef2-c0db-4fa7-9d91-95df203dff49" volumeName="kubernetes.io/empty-dir/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-tuned" seLinuxMountContext=""
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283662    4614 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 09:35:19.283799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.283667    4614 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 09:35:19.285623 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.284155    4614 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:35:19.286224 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.286200    4614 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:35:19.287153 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.287131    4614 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:35:19.289079 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.287562    4614 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 09:35:19.296284 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.296265    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/1.log"
Nov 02 09:35:19.296462 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.296268    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/1.log"
Nov 02 09:35:19.306755 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.306725    4614 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 09:35:19.308079 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.308060    4614 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 09:35:19.308079 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.308079    4614 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 09:35:19.308241 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.308104    4614 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 09:35:19.308241 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:19.308143    4614 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 09:35:19.312202 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.312106    4614 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 09:35:19.331475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.331452    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/1.log"
Nov 02 09:35:19.331587 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.331479    4614 generic.go:356] "Generic (PLEG): container finished" podID="2d28887e-4aa9-4209-a51a-fec831136660" containerID="ce9276cd5977d158006454c6d5076ffdb908f3171c697d08603d28e20fa97374" exitCode=255
Nov 02 09:35:19.346084 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.346048    4614 generic.go:356] "Generic (PLEG): container finished" podID="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" containerID="4680a323d324a457e2e27fc4e876588ba1515af257b4b410001ce85e951230d4" exitCode=0
Nov 02 09:35:19.346084 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.346070    4614 generic.go:356] "Generic (PLEG): container finished" podID="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" containerID="e099aba2156a917faa484c556c599578fec2ab59ae9f47d247dbe23ecbf94676" exitCode=0
Nov 02 09:35:19.346084 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.346077    4614 generic.go:356] "Generic (PLEG): container finished" podID="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" containerID="02dbae894a922a02f9defca24ee501c916752da1d483975d009b58a68a43ddb4" exitCode=0
Nov 02 09:35:19.346084 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.346084    4614 generic.go:356] "Generic (PLEG): container finished" podID="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" containerID="08d5d33f2a07c85f1d08fe4fdf61a12a3d35b547784bd7cb0f75ac8e2b8cd765" exitCode=0
Nov 02 09:35:19.346084 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.346090    4614 generic.go:356] "Generic (PLEG): container finished" podID="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" containerID="4d08032f9df97d5ae625422e9277c2cb4c734af3d38dbc6cd7c758ebe5917915" exitCode=0
Nov 02 09:35:19.346481 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.346095    4614 generic.go:356] "Generic (PLEG): container finished" podID="f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb" containerID="2b5b08e1e86bd1c3ce31acde64eedfe65c7e82be9b6fd013c087143a6f59225c" exitCode=0
Nov 02 09:35:19.350649 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.350624    4614 generic.go:356] "Generic (PLEG): container finished" podID="33ba6db318717035e56c8b1001fb10e6" containerID="13b4563108d0a935f80344378aa8b08036712f42972aaa187e5d448253a33f87" exitCode=0
Nov 02 09:35:19.369755 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.369690    4614 manager.go:324] Recovery completed
Nov 02 09:35:19.390157 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.390135    4614 cpu_manager.go:226] "Starting" policy="none"
Nov 02 09:35:19.390157 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.390152    4614 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 09:35:19.390289 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.390172    4614 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 09:35:19.390323 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.390290    4614 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 09:35:19.390323 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.390300    4614 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 09:35:19.390399 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.390328    4614 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 09:35:19.390399 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.390334    4614 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 09:35:19.390399 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.390339    4614 policy_none.go:50] "Start"
Nov 02 09:35:19.390399 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.390348    4614 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 09:35:19.390399 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.390370    4614 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 09:35:19.390529 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.390490    4614 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 09:35:19.390529 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.390496    4614 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 09:35:19.390529 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.390508    4614 policy_none.go:44] "Start"
Nov 02 09:35:19.395043 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.395006    4614 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 09:35:19.397994 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.397982    4614 manager.go:341] "Starting Device Plugin manager"
Nov 02 09:35:19.398528 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:19.398007    4614 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 09:35:19.398581 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.398530    4614 server.go:87] "Starting device plugin registration server"
Nov 02 09:35:19.398765 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.398756    4614 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 09:35:19.398806 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.398767    4614 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 09:35:19.398869 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.398853    4614 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 09:35:19.399000 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.398961    4614 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 09:35:19.399000 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.399009    4614 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 09:35:19.399155 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.399015    4614 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 09:35:19.399155 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.399071    4614 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T09:35:19.398994618Z","UUID":"ea9c0a53-4b1e-4646-a1dc-059e8e12d22d","Handler":null,"Name":"","Endpoint":""}
Nov 02 09:35:19.400324 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.400304    4614 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 09:35:19.400421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.400334    4614 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 09:35:19.400574 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:19.400557    4614 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 09:35:19.408295 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.408251    4614 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"]
Nov 02 09:35:19.409426 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.409378    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal" event={"ID":"a2d480d50be16b70f16fe76e324fa19d","Type":"ContainerStarted","Data":"4a46a25b79cd4c2cb009f709431733c8f0c1b08945fd748b08b96c07dedfe722"}
Nov 02 09:35:19.409501 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.409440    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal" event={"ID":"a2d480d50be16b70f16fe76e324fa19d","Type":"ContainerStarted","Data":"5378d6236e7ea1b301c2da5f0cd98cc06abd991f2cb580cba7d08b14abc5d3cc"}
Nov 02 09:35:19.409648 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.409616    4614 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.409772 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.409712    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal" event={"ID":"33ba6db318717035e56c8b1001fb10e6","Type":"ContainerStarted","Data":"08eb5d5b499605bdc5cb40f03db2de845424d95bc6163569725ebec910a96774"}
Nov 02 09:35:19.409772 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.409737    4614 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.409884 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.409787    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal" event={"ID":"33ba6db318717035e56c8b1001fb10e6","Type":"ContainerDied","Data":"13b4563108d0a935f80344378aa8b08036712f42972aaa187e5d448253a33f87"}
Nov 02 09:35:19.409884 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.409824    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal" event={"ID":"33ba6db318717035e56c8b1001fb10e6","Type":"ContainerStarted","Data":"6c0719828efa2d70fa740192c6b713ae871a0172206cc80593be9364ae330fba"}
Nov 02 09:35:19.423344 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.423320    4614 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 09:35:19.423453 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:19.423375    4614 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.423963 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.423938    4614 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 09:35:19.424042 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:19.423986    4614 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.488489 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.488461    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33ba6db318717035e56c8b1001fb10e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"33ba6db318717035e56c8b1001fb10e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.488601 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.488492    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33ba6db318717035e56c8b1001fb10e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"33ba6db318717035e56c8b1001fb10e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.488601 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.488512    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2d480d50be16b70f16fe76e324fa19d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"a2d480d50be16b70f16fe76e324fa19d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.499575 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.499556    4614 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 09:35:19.500869 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.500847    4614 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 09:35:19.500945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.500882    4614 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 09:35:19.500945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.500893    4614 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-174.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 09:35:19.500945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.500933    4614 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.513038 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.513019    4614 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.513138 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.513074    4614 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.589493 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.589466    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33ba6db318717035e56c8b1001fb10e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"33ba6db318717035e56c8b1001fb10e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.589493 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.589489    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33ba6db318717035e56c8b1001fb10e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"33ba6db318717035e56c8b1001fb10e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.589637 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.589539    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33ba6db318717035e56c8b1001fb10e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"33ba6db318717035e56c8b1001fb10e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.589637 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.589561    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2d480d50be16b70f16fe76e324fa19d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"a2d480d50be16b70f16fe76e324fa19d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.589637 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.589596    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2d480d50be16b70f16fe76e324fa19d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"a2d480d50be16b70f16fe76e324fa19d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:19.589637 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:19.589626    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33ba6db318717035e56c8b1001fb10e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal\" (UID: \"33ba6db318717035e56c8b1001fb10e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:20.261197 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.261165    4614 apiserver.go:52] "Watching apiserver"
Nov 02 09:35:20.282833 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.282806    4614 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 09:35:20.283579 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.283552    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-h6h6n","kube-system/global-pull-secret-syncer-lxn8z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-h5bzx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs","openshift-dns/node-resolver-zz4q2","openshift-image-registry/node-ca-xz8m5","openshift-console-operator/console-operator-7cd56b8dbc-n99gd","openshift-image-registry/image-registry-bc8fbd47b-96z8h","openshift-image-registry/image-registry-79578858d5-6w82j","openshift-insights/insights-runtime-extractor-nbfgg","openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw","openshift-network-console/networking-console-plugin-86c478bb88-qrb6h","openshift-network-diagnostics/network-check-source-6f78d59f99-mvxdw","kube-system/kas-connection-checker-685895c7c9-cf7t5","openshift-ingress-canary/ingress-canary-xkrcs","openshift-ovn-kubernetes/ovnkube-node-wtrqd","openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp","openshift-multus/network-metrics-daemon-v9xng","openshift-dns/dns-default-zgjvs","openshift-multus/multus-zhmvb","openshift-multus/multus-additional-cni-plugins-kmqdf","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sb5ns","openshift-ingress/router-default-68d88c448f-z5k5p","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd","openshift-cluster-node-tuning-operator/tuned-gfgcv","openshift-insights/insights-operator-7c4986d449-pdw8n","kube-system/kas-connection-checker-685895c7c9-pprd9","kube-system/konnectivity-agent-xzrll","kube-system/kas-connection-checker-685895c7c9-xtlc6"]
Nov 02 09:35:20.287335 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.287211    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:20.287815 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.287787    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.287993 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.287795    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.288712 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.288694    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.289051 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.289032    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.289216 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.289199    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fzpc9\""
Nov 02 09:35:20.289367 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.289329    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.289540 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.289521    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.289597 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.289587    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 09:35:20.289857 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.289839    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-q24jz\""
Nov 02 09:35:20.289977 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.289962    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jgpb4\""
Nov 02 09:35:20.290206 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.289997    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 09:35:20.290284 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.290033    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.291142 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.291121    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:20.291462 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.291445    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.291554 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.291447    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 09:35:20.291631 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.291613    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-66z6l\""
Nov 02 09:35:20.291732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.291719    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 09:35:20.291871 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.291842    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 09:35:20.292163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.292147    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.292460 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.292443    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.292537 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.292319    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 09:35:20.292616 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.292579    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.292616 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.292592    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 09:35:20.292727 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.292711    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.292875 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.292856    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:20.292947 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.292239    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 09:35:20.293150 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.293127    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 09:35:20.293150 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.293145    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bhljs\""
Nov 02 09:35:20.293772 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.293751    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.293900 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.293884    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.293958 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.293937    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 09:35:20.293958 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.293950    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.294405 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.294388    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:20.294972 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.294945    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 09:35:20.295324 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.295309    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hrc8h\""
Nov 02 09:35:20.296042 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.296011    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.296123 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.296063    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.296210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.296190    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-w8zs4\""
Nov 02 09:35:20.296594 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.296566    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 09:35:20.296781 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.296767    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:20.297744 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.297726    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cmxq6\""
Nov 02 09:35:20.298338 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.298318    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.298557 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.298537    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.298637 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.298622    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.298730 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.298714    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 09:35:20.298785 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.298746    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 09:35:20.298863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.298785    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-68wrf\""
Nov 02 09:35:20.298863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.298801    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 09:35:20.298926 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.298867    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 09:35:20.300115 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.300091    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 09:35:20.300201 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.300119    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kdt4x\""
Nov 02 09:35:20.300201 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.300098    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 09:35:20.300982 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.300963    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.301707 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.301690    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:20.302445 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.302424    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.302748    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.302807    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.302819    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.302847    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.302853    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.302867    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.302888    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.302928    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.302936    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.302942    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.302958    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.302847    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303004    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.302875    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 09:35:20.302768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303023    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 09:35:20.303830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303153    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.303830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303184    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 09:35:20.303830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303189    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 09:35:20.303830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303191    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 09:35:20.303830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303204    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 09:35:20.303830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303223    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 09:35:20.303830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303266    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.303830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303270    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.303830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303427    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.303830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303544    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bs7z2\""
Nov 02 09:35:20.303830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303594    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbhf4\""
Nov 02 09:35:20.303830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303633    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pcvs5\""
Nov 02 09:35:20.303830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303680    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 09:35:20.303830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303715    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qbp4s\""
Nov 02 09:35:20.304311 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303866    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zc5hs\""
Nov 02 09:35:20.304311 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.303884    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 09:35:20.304311 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.304055    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-llzhk\""
Nov 02 09:35:20.304311 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.304070    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t9b8s\""
Nov 02 09:35:20.304311 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.304078    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7vqld\""
Nov 02 09:35:20.304311 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.304059    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.304311 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.304107    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-982vl\""
Nov 02 09:35:20.304311 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.304215    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 09:35:20.304964 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.304940    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 09:35:20.305585 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.305567    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 09:35:20.306671 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.306648    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 09:35:20.306766 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.306752    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.306854 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.306842    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.307064 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307049    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9d4cl\""
Nov 02 09:35:20.307141 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307121    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 09:35:20.307226 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307212    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9tqmn\""
Nov 02 09:35:20.307278 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307235    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 09:35:20.307366 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307341    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 09:35:20.307454 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307439    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 09:35:20.307520 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307438    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f2j22\""
Nov 02 09:35:20.307571 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307556    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xrsh4\""
Nov 02 09:35:20.307619 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307598    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 09:35:20.307619 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307602    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sbddj\""
Nov 02 09:35:20.307721 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307614    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 09:35:20.307721 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307690    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 09:35:20.307832 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307722    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gztjj\""
Nov 02 09:35:20.307832 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307746    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 09:35:20.308104 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.308084    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 09:35:20.308242 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307691    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 09:35:20.308327 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.307692    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zwj6d\""
Nov 02 09:35:20.361018 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.360995    4614 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:20.361195 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.361079    4614 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:20.374834 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.374814    4614 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 09:35:20.376451 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.376433    4614 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 09:35:20.376498 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.376475    4614 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:20.377210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.377197    4614 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 09:35:20.377245 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.377228    4614 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-174.us-east-2.compute.internal"
Nov 02 09:35:20.393081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393059    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-systemd\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.393152 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393088    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-multus-certs\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.393152 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393114    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:20.393208 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393166    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2608084-c9df-4998-a521-fb2239b3c349-config\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:20.393246 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393202    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:20.393287 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393263    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:20.393321 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393297    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-run-netns\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.393382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393318    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmw7c\" (UniqueName: \"kubernetes.io/projected/b9ed0d41-1122-4ec2-a9af-e84cc71605df-kube-api-access-pmw7c\") pod \"volume-data-source-validator-58c9f5f46d-sb5ns\" (UID: \"b9ed0d41-1122-4ec2-a9af-e84cc71605df\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sb5ns"
Nov 02 09:35:20.393382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393348    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfscl\" (UniqueName: \"kubernetes.io/projected/d9ecca6d-3d39-4f10-9129-06feb4b292fa-kube-api-access-nfscl\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:35:20.393486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393425    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6qw7\" (UniqueName: \"kubernetes.io/projected/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-kube-api-access-l6qw7\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.393486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393464    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b36bfac6-a600-4029-a446-7de82575fb0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:20.393583 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393493    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7fg\" (UniqueName: \"kubernetes.io/projected/fce2663d-12a8-427f-b8ec-0d2238a06470-kube-api-access-qh7fg\") pod \"kas-connection-checker-685895c7c9-xtlc6\" (UID: \"fce2663d-12a8-427f-b8ec-0d2238a06470\") " pod="kube-system/kas-connection-checker-685895c7c9-xtlc6"
Nov 02 09:35:20.393583 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393520    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8htqg\" (UniqueName: \"kubernetes.io/projected/a05b1aec-cdf3-424b-8dfe-d4a6f06d6536-kube-api-access-8htqg\") pod \"network-check-source-6f78d59f99-mvxdw\" (UID: \"a05b1aec-cdf3-424b-8dfe-d4a6f06d6536\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvxdw"
Nov 02 09:35:20.393583 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393555    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysconfig\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.393583 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393565    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:20.393583 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393565    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2608084-c9df-4998-a521-fb2239b3c349-config\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:20.393792 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393583    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77530df7-545e-42b0-ad16-ce934cb10dab-host\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:35:20.393792 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393608    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5b95e7b-2ce0-45a1-9859-0d2378576064-dbus\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:20.393792 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393639    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85741a00-b702-46cb-a90c-e000bf9976f0-serving-cert\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:20.393792 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393675    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psmww\" (UniqueName: \"kubernetes.io/projected/78cde7e0-a2f3-4f33-ba88-6c9fedc723eb-kube-api-access-psmww\") pod \"kas-connection-checker-685895c7c9-cf7t5\" (UID: \"78cde7e0-a2f3-4f33-ba88-6c9fedc723eb\") " pod="kube-system/kas-connection-checker-685895c7c9-cf7t5"
Nov 02 09:35:20.393792 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393786    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfc6ef2-c0db-4fa7-9d91-95df203dff49-tmp\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.393973 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393824    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-cni-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.393973 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393844    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-bound-sa-token\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.393973 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393861    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfmg8\" (UniqueName: \"kubernetes.io/projected/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-kube-api-access-cfmg8\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.393973 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393900    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysctl-d\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.393973 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393912    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfc6ef2-c0db-4fa7-9d91-95df203dff49-tmp\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.393973 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393934    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/77530df7-545e-42b0-ad16-ce934cb10dab-serviceca\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:35:20.393973 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.393962    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85741a00-b702-46cb-a90c-e000bf9976f0-serving-cert\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:20.394193 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394041    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2608084-c9df-4998-a521-fb2239b3c349-serving-cert\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:20.394193 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394063    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-ovn\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.394193 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394080    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ms9\" (UniqueName: \"kubernetes.io/projected/2d28887e-4aa9-4209-a51a-fec831136660-kube-api-access-n8ms9\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:20.394193 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394097    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09b69cf9-00f2-48f9-9c66-f22027a2376c-cni-binary-copy\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.394193 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394119    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/77530df7-545e-42b0-ad16-ce934cb10dab-serviceca\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:35:20.394193 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394137    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-installation-pull-secrets\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.394193 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394174    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ql9\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-kube-api-access-v5ql9\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.394515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394203    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.394515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394211    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2608084-c9df-4998-a521-fb2239b3c349-serving-cert\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:20.394515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394230    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9tb2\" (UniqueName: \"kubernetes.io/projected/b36bfac6-a600-4029-a446-7de82575fb0a-kube-api-access-n9tb2\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:20.394515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394260    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-var-lib-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.394515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394285    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b36bfac6-a600-4029-a446-7de82575fb0a-data-volume\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:20.394515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394311    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:20.394515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394337    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cnibin\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.394515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394369    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09b69cf9-00f2-48f9-9c66-f22027a2376c-cni-binary-copy\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.394515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394382    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e5393080-e20a-47fc-8dc7-125f42457736-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:20.394515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394420    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvdcd\" (UniqueName: \"kubernetes.io/projected/95d7a9df-37e3-492b-9540-3df603931723-kube-api-access-qvdcd\") pod \"kas-connection-checker-685895c7c9-pprd9\" (UID: \"95d7a9df-37e3-492b-9540-3df603931723\") " pod="kube-system/kas-connection-checker-685895c7c9-pprd9"
Nov 02 09:35:20.394515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394452    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-cni-netd\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.394515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394485    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-cni-bin\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.394515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394493    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.394515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394503    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-installation-pull-secrets\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.395103 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394573    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zv58\" (UniqueName: \"kubernetes.io/projected/961f499a-2ed3-4e0d-b5d2-9194d18b696a-kube-api-access-2zv58\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:20.395103 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394620    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b36bfac6-a600-4029-a446-7de82575fb0a-data-volume\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:20.395103 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394662    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-node-log\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.395103 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394712    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e5393080-e20a-47fc-8dc7-125f42457736-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:20.395103 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394716    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-trusted-ca\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.395103 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394832    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-kubelet\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.395103 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394860    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-image-registry-private-configuration\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.395103 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394882    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.395103 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.394978    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-certificates\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.395103 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395007    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-trusted-ca\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.395103 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395013    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:20.395103 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395047    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-modprobe-d\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.395103 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395095    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-image-registry-private-configuration\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395112    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395120    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-certificates\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395132    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395150    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502f9765-e116-4359-b513-c1d00c90d7de-tmp-dir\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395164    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-registration-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395193    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e8df50bb-8d26-42c1-b496-c9d9f56785d9-konnectivity-ca\") pod \"konnectivity-agent-xzrll\" (UID: \"e8df50bb-8d26-42c1-b496-c9d9f56785d9\") " pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395268    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbgj\" (UniqueName: \"kubernetes.io/projected/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-kube-api-access-zsbgj\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395286    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502f9765-e116-4359-b513-c1d00c90d7de-tmp-dir\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395308    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-tuned\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395386    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395392    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-tuned\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395402    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e8df50bb-8d26-42c1-b496-c9d9f56785d9-konnectivity-ca\") pod \"konnectivity-agent-xzrll\" (UID: \"e8df50bb-8d26-42c1-b496-c9d9f56785d9\") " pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395423    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-certificates\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395451    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85741a00-b702-46cb-a90c-e000bf9976f0-snapshots\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395476    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbxf5\" (UniqueName: \"kubernetes.io/projected/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-kube-api-access-bbxf5\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395532    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:20.395560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395560    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5b95e7b-2ce0-45a1-9859-0d2378576064-kubelet-config\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395610    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5b95e7b-2ce0-45a1-9859-0d2378576064-original-pull-secret\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395626    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d28887e-4aa9-4209-a51a-fec831136660-config\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395641    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85741a00-b702-46cb-a90c-e000bf9976f0-snapshots\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395652    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-certificates\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395654    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-system-cni-dir\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395693    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395722    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-systemd-units\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395749    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhsmx\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-kube-api-access-xhsmx\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395802    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz5dm\" (UniqueName: \"kubernetes.io/projected/09b69cf9-00f2-48f9-9c66-f22027a2376c-kube-api-access-xz5dm\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395831    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395860    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395864    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d28887e-4aa9-4209-a51a-fec831136660-config\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395884    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-sys\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395889    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395919    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-installation-pull-secrets\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395954    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.395981    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-netns\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396025    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85741a00-b702-46cb-a90c-e000bf9976f0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396039    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396076    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbd4d998-a5be-4a0a-9113-8f3782df753a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396084    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-installation-pull-secrets\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396108    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-etc-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396200    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-ovnkube-script-lib\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396265    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbd4d998-a5be-4a0a-9113-8f3782df753a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396279    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396309    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfq4v\" (UniqueName: \"kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v\") pod \"network-check-target-h5bzx\" (UID: \"dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e\") " pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:35:20.396333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396341    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85741a00-b702-46cb-a90c-e000bf9976f0-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396378    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-lib-modules\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396413    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4zq4\" (UniqueName: \"kubernetes.io/projected/80746de3-f9f3-4efe-a963-9b37f290fcb7-kube-api-access-q4zq4\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396452    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzrxx\" (UniqueName: \"kubernetes.io/projected/a2608084-c9df-4998-a521-fb2239b3c349-kube-api-access-zzrxx\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396474    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-ovnkube-script-lib\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396479    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-socket-dir-parent\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396503    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-k8s-cni-cncf-io\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396531    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396562    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86xb\" (UniqueName: \"kubernetes.io/projected/dbd4d998-a5be-4a0a-9113-8f3782df753a-kube-api-access-b86xb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396653    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-stats-auth\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396672    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396693    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-device-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396710    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-kubernetes\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396730    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wmsb\" (UniqueName: \"kubernetes.io/projected/adfc6ef2-c0db-4fa7-9d91-95df203dff49-kube-api-access-7wmsb\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396770    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396854    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-stats-auth\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396897    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwtd7\" (UniqueName: \"kubernetes.io/projected/118b62a2-be2c-4119-87d4-914839ecc25a-kube-api-access-xwtd7\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.396949    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/502f9765-e116-4359-b513-c1d00c90d7de-config-volume\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397003    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b25xs\" (UniqueName: \"kubernetes.io/projected/85741a00-b702-46cb-a90c-e000bf9976f0-kube-api-access-b25xs\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397050    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397078    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397083    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/502f9765-e116-4359-b513-c1d00c90d7de-config-volume\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397104    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-sys-fs\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397131    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbd4d998-a5be-4a0a-9113-8f3782df753a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397206    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3c01c082-8479-460d-a80d-5e590ab2fcbe-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397237    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-log-socket\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397254    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-env-overrides\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397288    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d28887e-4aa9-4209-a51a-fec831136660-serving-cert\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:20.397279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397287    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbd4d998-a5be-4a0a-9113-8f3782df753a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397386    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3603dee5-a95f-47e4-9d60-381e82b8afa4-ca-trust-extracted\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397410    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-env-overrides\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397465    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d28887e-4aa9-4209-a51a-fec831136660-serving-cert\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397487    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-hosts-file\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397497    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3603dee5-a95f-47e4-9d60-381e82b8afa4-ca-trust-extracted\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397524    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e8df50bb-8d26-42c1-b496-c9d9f56785d9-agent-certs\") pod \"konnectivity-agent-xzrll\" (UID: \"e8df50bb-8d26-42c1-b496-c9d9f56785d9\") " pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397541    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9ecca6d-3d39-4f10-9129-06feb4b292fa-iptables-alerter-script\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397524    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3c01c082-8479-460d-a80d-5e590ab2fcbe-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397595    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26adcc96-ef0b-416f-8b15-e01be59fb5ae-cert\") pod \"ingress-canary-xkrcs\" (UID: \"26adcc96-ef0b-416f-8b15-e01be59fb5ae\") " pod="openshift-ingress-canary/ingress-canary-xkrcs"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397654    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6kp\" (UniqueName: \"kubernetes.io/projected/26adcc96-ef0b-416f-8b15-e01be59fb5ae-kube-api-access-zr6kp\") pod \"ingress-canary-xkrcs\" (UID: \"26adcc96-ef0b-416f-8b15-e01be59fb5ae\") " pod="openshift-ingress-canary/ingress-canary-xkrcs"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397734    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e8df50bb-8d26-42c1-b496-c9d9f56785d9-agent-certs\") pod \"konnectivity-agent-xzrll\" (UID: \"e8df50bb-8d26-42c1-b496-c9d9f56785d9\") " pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397747    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9ecca6d-3d39-4f10-9129-06feb4b292fa-iptables-alerter-script\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397765    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-kubelet\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397785    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-daemon-config\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397821    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2daf8a59-8027-42b4-8414-55a0883bc04f-ca-trust-extracted\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397896    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-systemd\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397911    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26adcc96-ef0b-416f-8b15-e01be59fb5ae-cert\") pod \"ingress-canary-xkrcs\" (UID: \"26adcc96-ef0b-416f-8b15-e01be59fb5ae\") " pod="openshift-ingress-canary/ingress-canary-xkrcs"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397931    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-daemon-config\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397938    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2daf8a59-8027-42b4-8414-55a0883bc04f-ca-trust-extracted\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.397985    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-hostroot\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398018    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85741a00-b702-46cb-a90c-e000bf9976f0-tmp\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398053    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-conf-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398097    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85741a00-b702-46cb-a90c-e000bf9976f0-tmp\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398143    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d28887e-4aa9-4209-a51a-fec831136660-trusted-ca\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398206    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b36bfac6-a600-4029-a446-7de82575fb0a-crio-socket\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398230    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-var-lib-kubelet\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398259    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-run\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398293    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b2xg\" (UniqueName: \"kubernetes.io/projected/3c01c082-8479-460d-a80d-5e590ab2fcbe-kube-api-access-7b2xg\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398327    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/118b62a2-be2c-4119-87d4-914839ecc25a-ovn-node-metrics-cert\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398343    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-trusted-ca\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398391    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-socket-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398422    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqvqm\" (UniqueName: \"kubernetes.io/projected/77530df7-545e-42b0-ad16-ce934cb10dab-kube-api-access-pqvqm\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398480    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-slash\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398507    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d28887e-4aa9-4209-a51a-fec831136660-trusted-ca\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398511    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/118b62a2-be2c-4119-87d4-914839ecc25a-ovn-node-metrics-cert\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398528    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-image-registry-private-configuration\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398647    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-system-cni-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398684    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-cnibin\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398701    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-cni-multus\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398719    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-tmp-dir\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398726    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-trusted-ca\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398734    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-ovnkube-config\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398724    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-image-registry-private-configuration\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398766    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-bound-sa-token\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.398770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398789    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-cni-bin\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.400025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398799    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-tmp-dir\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:35:20.400025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398822    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-etc-kubernetes\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.400025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398905    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgmvv\" (UniqueName: \"kubernetes.io/projected/502f9765-e116-4359-b513-c1d00c90d7de-kube-api-access-jgmvv\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:20.400025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.398919    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/118b62a2-be2c-4119-87d4-914839ecc25a-ovnkube-config\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.400025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.399019    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85741a00-b702-46cb-a90c-e000bf9976f0-service-ca-bundle\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:20.400025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.399054    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysctl-conf\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.400025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.399135    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-host\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.400025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.399138    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85741a00-b702-46cb-a90c-e000bf9976f0-service-ca-bundle\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:20.400025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.399153    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9ecca6d-3d39-4f10-9129-06feb4b292fa-host-slash\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:35:20.400025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.399183    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-default-certificate\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:20.400025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.399425    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-default-certificate\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:20.403521 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.403500    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7fg\" (UniqueName: \"kubernetes.io/projected/fce2663d-12a8-427f-b8ec-0d2238a06470-kube-api-access-qh7fg\") pod \"kas-connection-checker-685895c7c9-xtlc6\" (UID: \"fce2663d-12a8-427f-b8ec-0d2238a06470\") " pod="kube-system/kas-connection-checker-685895c7c9-xtlc6"
Nov 02 09:35:20.403783 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.403764    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmw7c\" (UniqueName: \"kubernetes.io/projected/b9ed0d41-1122-4ec2-a9af-e84cc71605df-kube-api-access-pmw7c\") pod \"volume-data-source-validator-58c9f5f46d-sb5ns\" (UID: \"b9ed0d41-1122-4ec2-a9af-e84cc71605df\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-sb5ns"
Nov 02 09:35:20.403910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.403893    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6qw7\" (UniqueName: \"kubernetes.io/projected/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-kube-api-access-l6qw7\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.403986 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.403973    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-bound-sa-token\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.404274 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.404259    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psmww\" (UniqueName: \"kubernetes.io/projected/78cde7e0-a2f3-4f33-ba88-6c9fedc723eb-kube-api-access-psmww\") pod \"kas-connection-checker-685895c7c9-cf7t5\" (UID: \"78cde7e0-a2f3-4f33-ba88-6c9fedc723eb\") " pod="kube-system/kas-connection-checker-685895c7c9-cf7t5"
Nov 02 09:35:20.415693 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.415640    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbxf5\" (UniqueName: \"kubernetes.io/projected/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-kube-api-access-bbxf5\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:35:20.416312 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.416283    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfscl\" (UniqueName: \"kubernetes.io/projected/d9ecca6d-3d39-4f10-9129-06feb4b292fa-kube-api-access-nfscl\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:35:20.416546 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.416522    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfmg8\" (UniqueName: \"kubernetes.io/projected/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-kube-api-access-cfmg8\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.416620 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.416547    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ql9\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-kube-api-access-v5ql9\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.416673 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.416626    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8htqg\" (UniqueName: \"kubernetes.io/projected/a05b1aec-cdf3-424b-8dfe-d4a6f06d6536-kube-api-access-8htqg\") pod \"network-check-source-6f78d59f99-mvxdw\" (UID: \"a05b1aec-cdf3-424b-8dfe-d4a6f06d6536\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-mvxdw"
Nov 02 09:35:20.417812 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.417789    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8ms9\" (UniqueName: \"kubernetes.io/projected/2d28887e-4aa9-4209-a51a-fec831136660-kube-api-access-n8ms9\") pod \"console-operator-7cd56b8dbc-n99gd\" (UID: \"2d28887e-4aa9-4209-a51a-fec831136660\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:20.417998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.417984    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9tb2\" (UniqueName: \"kubernetes.io/projected/b36bfac6-a600-4029-a446-7de82575fb0a-kube-api-access-n9tb2\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:20.418054 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.417983    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zv58\" (UniqueName: \"kubernetes.io/projected/961f499a-2ed3-4e0d-b5d2-9194d18b696a-kube-api-access-2zv58\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:20.418154 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.418135    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvdcd\" (UniqueName: \"kubernetes.io/projected/95d7a9df-37e3-492b-9540-3df603931723-kube-api-access-qvdcd\") pod \"kas-connection-checker-685895c7c9-pprd9\" (UID: \"95d7a9df-37e3-492b-9540-3df603931723\") " pod="kube-system/kas-connection-checker-685895c7c9-pprd9"
Nov 02 09:35:20.418238 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.418224    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhsmx\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-kube-api-access-xhsmx\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.421393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.421370    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgmvv\" (UniqueName: \"kubernetes.io/projected/502f9765-e116-4359-b513-c1d00c90d7de-kube-api-access-jgmvv\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:20.421613 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.421592    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86xb\" (UniqueName: \"kubernetes.io/projected/dbd4d998-a5be-4a0a-9113-8f3782df753a-kube-api-access-b86xb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd\" (UID: \"dbd4d998-a5be-4a0a-9113-8f3782df753a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd"
Nov 02 09:35:20.421697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.421683    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzrxx\" (UniqueName: \"kubernetes.io/projected/a2608084-c9df-4998-a521-fb2239b3c349-kube-api-access-zzrxx\") pod \"service-ca-operator-544d749c8c-cwfdw\" (UID: \"a2608084-c9df-4998-a521-fb2239b3c349\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw"
Nov 02 09:35:20.421940 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.421921    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b2xg\" (UniqueName: \"kubernetes.io/projected/3c01c082-8479-460d-a80d-5e590ab2fcbe-kube-api-access-7b2xg\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:20.422825 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.422482    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b25xs\" (UniqueName: \"kubernetes.io/projected/85741a00-b702-46cb-a90c-e000bf9976f0-kube-api-access-b25xs\") pod \"insights-operator-7c4986d449-pdw8n\" (UID: \"85741a00-b702-46cb-a90c-e000bf9976f0\") " pod="openshift-insights/insights-operator-7c4986d449-pdw8n"
Nov 02 09:35:20.422825 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.422517    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqvqm\" (UniqueName: \"kubernetes.io/projected/77530df7-545e-42b0-ad16-ce934cb10dab-kube-api-access-pqvqm\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:35:20.422825 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.422595    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wmsb\" (UniqueName: \"kubernetes.io/projected/adfc6ef2-c0db-4fa7-9d91-95df203dff49-kube-api-access-7wmsb\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.422825 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.422769    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz5dm\" (UniqueName: \"kubernetes.io/projected/09b69cf9-00f2-48f9-9c66-f22027a2376c-kube-api-access-xz5dm\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.423113 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.422921    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwtd7\" (UniqueName: \"kubernetes.io/projected/118b62a2-be2c-4119-87d4-914839ecc25a-kube-api-access-xwtd7\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.423113 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.422941    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-bound-sa-token\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.423214 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.423158    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfq4v\" (UniqueName: \"kubernetes.io/projected/dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e-kube-api-access-jfq4v\") pod \"network-check-target-h5bzx\" (UID: \"dfd65a5e-d4eb-4e5a-8a99-e3431b306c4e\") " pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:35:20.423269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.423249    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6kp\" (UniqueName: \"kubernetes.io/projected/26adcc96-ef0b-416f-8b15-e01be59fb5ae-kube-api-access-zr6kp\") pod \"ingress-canary-xkrcs\" (UID: \"26adcc96-ef0b-416f-8b15-e01be59fb5ae\") " pod="openshift-ingress-canary/ingress-canary-xkrcs"
Nov 02 09:35:20.423459 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.423444    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbgj\" (UniqueName: \"kubernetes.io/projected/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-kube-api-access-zsbgj\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:20.423585 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.423570    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4zq4\" (UniqueName: \"kubernetes.io/projected/80746de3-f9f3-4efe-a963-9b37f290fcb7-kube-api-access-q4zq4\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:20.499930 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.499891    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-etc-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.500100 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.499955    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:20.500100 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.499980    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-lib-modules\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.500100 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.499998    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-etc-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.500100 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500002    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-socket-dir-parent\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.500100 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500078    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-socket-dir-parent\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.500100 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500081    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-k8s-cni-cncf-io\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.500106    4614 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500115    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.500121    4614 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79578858d5-6w82j: secret "image-registry-tls" not found
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500135    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500157    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-device-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.500199    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls podName:3603dee5-a95f-47e4-9d60-381e82b8afa4 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:21.000176816 +0000 UTC m=+1.832834895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls") pod "image-registry-79578858d5-6w82j" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4") : secret "image-registry-tls" not found
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500232    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-lib-modules\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500268    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500276    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-kubernetes\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500314    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500320    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-k8s-cni-cncf-io\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500330    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-device-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500278    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500344    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500398    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-kubernetes\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500420    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.500421 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.500431    4614 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.500448    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:21.000434297 +0000 UTC m=+1.833092388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : configmap references non-existent config key: service-ca.crt
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500465    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-sys-fs\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500464    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.500489    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:21.00047772 +0000 UTC m=+1.833135777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : secret "router-metrics-certs-default" not found
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500504    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-log-socket\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500508    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-sys-fs\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500532    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-hosts-file\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500551    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-log-socket\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500559    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-kubelet\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500586    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-systemd\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500619    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-hostroot\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500635    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-kubelet\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500648    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-systemd\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500650    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-conf-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500653    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-hostroot\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500671    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-conf-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500701    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92f5b78b-58a0-4d0d-9bf6-97c4943597fd-hosts-file\") pod \"node-resolver-zz4q2\" (UID: \"92f5b78b-58a0-4d0d-9bf6-97c4943597fd\") " pod="openshift-dns/node-resolver-zz4q2"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500729    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b36bfac6-a600-4029-a446-7de82575fb0a-crio-socket\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500744    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b36bfac6-a600-4029-a446-7de82575fb0a-crio-socket\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500757    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-var-lib-kubelet\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500779    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-run\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500805    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-socket-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500831    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-slash\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500851    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-run\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500856    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-system-cni-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500860    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-socket-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500872    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-cnibin\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500876    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-slash\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500888    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-cni-multus\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500902    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-var-lib-kubelet\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500913    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-system-cni-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500907    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-cni-bin\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500923    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-cnibin\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500928    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-cni-bin\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500938    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-cni-multus\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500940    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-etc-kubernetes\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500967    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysctl-conf\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500982    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-etc-kubernetes\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.500988    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-host\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501014    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9ecca6d-3d39-4f10-9129-06feb4b292fa-host-slash\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501038    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-systemd\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501078    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-multus-certs\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501089    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9ecca6d-3d39-4f10-9129-06feb4b292fa-host-slash\") pod \"iptables-alerter-h6h6n\" (UID: \"d9ecca6d-3d39-4f10-9129-06feb4b292fa\") " pod="openshift-network-operator/iptables-alerter-h6h6n"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501080    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysctl-conf\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501104    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-systemd\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501112    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-host\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501120    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-multus-certs\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501119    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501153    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:20.501137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501169    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-run-netns\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.501177    4614 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501191    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-run-netns\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501194    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b36bfac6-a600-4029-a446-7de82575fb0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.501210    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls podName:3c01c082-8479-460d-a80d-5e590ab2fcbe nodeName:}" failed. No retries permitted until 2026-11-02 09:35:21.001200497 +0000 UTC m=+1.833858560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mlxbp" (UID: "3c01c082-8479-460d-a80d-5e590ab2fcbe") : secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501236    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysconfig\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.501243    4614 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501251    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77530df7-545e-42b0-ad16-ce934cb10dab-host\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501268    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5b95e7b-2ce0-45a1-9859-0d2378576064-dbus\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501283    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysconfig\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501285    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77530df7-545e-42b0-ad16-ce934cb10dab-host\") pod \"node-ca-xz8m5\" (UID: \"77530df7-545e-42b0-ad16-ce934cb10dab\") " pod="openshift-image-registry/node-ca-xz8m5"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.501299    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls podName:5e2f49f4-2884-4804-bb3a-3ee463eeb4f8 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:21.001285628 +0000 UTC m=+1.833943700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ntl2c" (UID: "5e2f49f4-2884-4804-bb3a-3ee463eeb4f8") : secret "samples-operator-tls" not found
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501318    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5b95e7b-2ce0-45a1-9859-0d2378576064-dbus\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501316    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-cni-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501348    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysctl-d\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501374    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-multus-cni-dir\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501381    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-ovn\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501404    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-var-lib-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501420    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501418    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-ovn\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501439    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cnibin\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501460    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-sysctl-d\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501449    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-var-lib-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.501481    4614 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501474    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-cnibin\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501503    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-cni-netd\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501479    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-cni-netd\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.501516    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs podName:961f499a-2ed3-4e0d-b5d2-9194d18b696a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:21.001506925 +0000 UTC m=+1.834165007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs") pod "network-metrics-daemon-v9xng" (UID: "961f499a-2ed3-4e0d-b5d2-9194d18b696a") : secret "metrics-daemon-secret" not found
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501543    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-cni-bin\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501567    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-node-log\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501568    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-cni-bin\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501587    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-kubelet\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501602    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-node-log\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501608    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501619    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-var-lib-kubelet\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501638    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.501677    4614 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501671    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-modprobe-d\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501708    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501709    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b36bfac6-a600-4029-a446-7de82575fb0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.501688    4614 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8fbd47b-96z8h: secret "image-registry-tls" not found
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501733    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501756    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-etc-modprobe-d\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501759    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-run-openvswitch\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.501772    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls podName:2daf8a59-8027-42b4-8414-55a0883bc04f nodeName:}" failed. No retries permitted until 2026-11-02 09:35:21.001758453 +0000 UTC m=+1.834416534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls") pod "image-registry-bc8fbd47b-96z8h" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f") : secret "image-registry-tls" not found
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.501712    4614 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501787    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-registration-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501807    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501758    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-registration-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.501813    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls podName:502f9765-e116-4359-b513-c1d00c90d7de nodeName:}" failed. No retries permitted until 2026-11-02 09:35:21.001801839 +0000 UTC m=+1.834459915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls") pod "dns-default-zgjvs" (UID: "502f9765-e116-4359-b513-c1d00c90d7de") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501850    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501873    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5b95e7b-2ce0-45a1-9859-0d2378576064-kubelet-config\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501889    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-system-cni-dir\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501908    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-systemd-units\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.501914    4614 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501933    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.501952    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert podName:e5393080-e20a-47fc-8dc7-125f42457736 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:21.001940422 +0000 UTC m=+1.834598483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qrb6h" (UID: "e5393080-e20a-47fc-8dc7-125f42457736") : secret "networking-console-plugin-cert" not found
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501957    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb-system-cni-dir\") pod \"multus-additional-cni-plugins-kmqdf\" (UID: \"f8ff18f0-d0af-43a6-a5b6-ab7d2dd48deb\") " pod="openshift-multus/multus-additional-cni-plugins-kmqdf"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501964    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/118b62a2-be2c-4119-87d4-914839ecc25a-systemd-units\") pod \"ovnkube-node-wtrqd\" (UID: \"118b62a2-be2c-4119-87d4-914839ecc25a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501977    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48aad7e1-623f-4eab-8a57-a50fefe7f2b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gfbs\" (UID: \"48aad7e1-623f-4eab-8a57-a50fefe7f2b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gfbs"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501979    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-sys\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.501993    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5b95e7b-2ce0-45a1-9859-0d2378576064-kubelet-config\") pod \"global-pull-secret-syncer-lxn8z\" (UID: \"c5b95e7b-2ce0-45a1-9859-0d2378576064\") " pod="kube-system/global-pull-secret-syncer-lxn8z"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.502008    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.502021    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adfc6ef2-c0db-4fa7-9d91-95df203dff49-sys\") pod \"tuned-gfgcv\" (UID: \"adfc6ef2-c0db-4fa7-9d91-95df203dff49\") " pod="openshift-cluster-node-tuning-operator/tuned-gfgcv"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.502032    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-netns\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.502054    4614 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.502063    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b69cf9-00f2-48f9-9c66-f22027a2376c-host-run-netns\") pod \"multus-zhmvb\" (UID: \"09b69cf9-00f2-48f9-9c66-f22027a2376c\") " pod="openshift-multus/multus-zhmvb"
Nov 02 09:35:20.502910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:20.502095    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls podName:b36bfac6-a600-4029-a446-7de82575fb0a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:21.002083195 +0000 UTC m=+1.834741253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nbfgg" (UID: "b36bfac6-a600-4029-a446-7de82575fb0a") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:20.591179 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:20.591147    4614 scope.go:122] "RemoveContainer" containerID="ce9276cd5977d158006454c6d5076ffdb908f3171c697d08603d28e20fa97374"
Nov 02 09:35:21.006775 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.006595    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:21.007071 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.006797    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:21.007071 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.006825    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:21.007071 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.006749    4614 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 09:35:21.007071 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.006899    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs podName:961f499a-2ed3-4e0d-b5d2-9194d18b696a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:22.006884586 +0000 UTC m=+2.839542644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs") pod "network-metrics-daemon-v9xng" (UID: "961f499a-2ed3-4e0d-b5d2-9194d18b696a") : secret "metrics-daemon-secret" not found
Nov 02 09:35:21.007071 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.006918    4614 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:21.007071 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.006940    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:21.007071 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.006973    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls podName:502f9765-e116-4359-b513-c1d00c90d7de nodeName:}" failed. No retries permitted until 2026-11-02 09:35:22.006957862 +0000 UTC m=+2.839615919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls") pod "dns-default-zgjvs" (UID: "502f9765-e116-4359-b513-c1d00c90d7de") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:21.007071 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.006994    4614 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 09:35:21.007071 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.006919    4614 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:21.007071 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007018    4614 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8fbd47b-96z8h: secret "image-registry-tls" not found
Nov 02 09:35:21.007071 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007025    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert podName:e5393080-e20a-47fc-8dc7-125f42457736 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:22.007015895 +0000 UTC m=+2.839673971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qrb6h" (UID: "e5393080-e20a-47fc-8dc7-125f42457736") : secret "networking-console-plugin-cert" not found
Nov 02 09:35:21.007071 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007059    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls podName:2daf8a59-8027-42b4-8414-55a0883bc04f nodeName:}" failed. No retries permitted until 2026-11-02 09:35:22.007047026 +0000 UTC m=+2.839705099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls") pod "image-registry-bc8fbd47b-96z8h" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f") : secret "image-registry-tls" not found
Nov 02 09:35:21.007071 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.007015    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:21.007071 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007071    4614 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.007104    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007147    4614 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.007147    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007157    4614 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79578858d5-6w82j: secret "image-registry-tls" not found
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007178    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls podName:b36bfac6-a600-4029-a446-7de82575fb0a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:22.007160463 +0000 UTC m=+2.839818531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nbfgg" (UID: "b36bfac6-a600-4029-a446-7de82575fb0a") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007203    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls podName:3603dee5-a95f-47e4-9d60-381e82b8afa4 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:22.007196566 +0000 UTC m=+2.839854624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls") pod "image-registry-79578858d5-6w82j" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4") : secret "image-registry-tls" not found
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007213    4614 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.007237    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007248    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:22.007237311 +0000 UTC m=+2.839895376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : secret "router-metrics-certs-default" not found
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007304    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:22.007295442 +0000 UTC m=+2.839953517 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : configmap references non-existent config key: service-ca.crt
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.007323    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.007371    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007415    4614 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007451    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls podName:3c01c082-8479-460d-a80d-5e590ab2fcbe nodeName:}" failed. No retries permitted until 2026-11-02 09:35:22.007443283 +0000 UTC m=+2.840101341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mlxbp" (UID: "3c01c082-8479-460d-a80d-5e590ab2fcbe") : secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007468    4614 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 09:35:21.007617 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.007497    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls podName:5e2f49f4-2884-4804-bb3a-3ee463eeb4f8 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:22.00748796 +0000 UTC m=+2.840146027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ntl2c" (UID: "5e2f49f4-2884-4804-bb3a-3ee463eeb4f8") : secret "samples-operator-tls" not found
Nov 02 09:35:21.366651 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.366583    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/2.log"
Nov 02 09:35:21.367472 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.367024    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/1.log"
Nov 02 09:35:21.367472 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.367065    4614 generic.go:356] "Generic (PLEG): container finished" podID="2d28887e-4aa9-4209-a51a-fec831136660" containerID="f21e613441812ae795f21576ceb5d8db17b26ad841895c573c1ebb0b76be8f6d" exitCode=255
Nov 02 09:35:21.367472 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.367142    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd" event={"ID":"2d28887e-4aa9-4209-a51a-fec831136660","Type":"ContainerDied","Data":"f21e613441812ae795f21576ceb5d8db17b26ad841895c573c1ebb0b76be8f6d"}
Nov 02 09:35:21.367472 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.367196    4614 scope.go:122] "RemoveContainer" containerID="ce9276cd5977d158006454c6d5076ffdb908f3171c697d08603d28e20fa97374"
Nov 02 09:35:21.367845 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:21.367814    4614 scope.go:122] "RemoveContainer" containerID="f21e613441812ae795f21576ceb5d8db17b26ad841895c573c1ebb0b76be8f6d"
Nov 02 09:35:21.368027 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:21.368008    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n99gd_openshift-console-operator(2d28887e-4aa9-4209-a51a-fec831136660)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd" podUID="2d28887e-4aa9-4209-a51a-fec831136660"
Nov 02 09:35:22.019337 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:22.019304    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:22.019337 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:22.019339    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:22.019589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:22.019403    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:22.019589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:22.019422    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:22.019589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:22.019450    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:22.019589 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019451    4614 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 09:35:22.019589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:22.019473    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:22.019589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:22.019501    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:22.019589 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019522    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.019502313 +0000 UTC m=+4.852160388 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : secret "router-metrics-certs-default" not found
Nov 02 09:35:22.019589 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019536    4614 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:22.019589 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019558    4614 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 09:35:22.019589 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019566    4614 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:22.019589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:22.019573    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:22.019589 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019582    4614 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8fbd47b-96z8h: secret "image-registry-tls" not found
Nov 02 09:35:22.019589 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019590    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls podName:3c01c082-8479-460d-a80d-5e590ab2fcbe nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.019572325 +0000 UTC m=+4.852230398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mlxbp" (UID: "3c01c082-8479-460d-a80d-5e590ab2fcbe") : secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019537    4614 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019622    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs podName:961f499a-2ed3-4e0d-b5d2-9194d18b696a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.019612071 +0000 UTC m=+4.852270150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs") pod "network-metrics-daemon-v9xng" (UID: "961f499a-2ed3-4e0d-b5d2-9194d18b696a") : secret "metrics-daemon-secret" not found
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019641    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls podName:2daf8a59-8027-42b4-8414-55a0883bc04f nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.019633152 +0000 UTC m=+4.852291213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls") pod "image-registry-bc8fbd47b-96z8h" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f") : secret "image-registry-tls" not found
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019644    4614 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019661    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.019652397 +0000 UTC m=+4.852310472 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : configmap references non-existent config key: service-ca.crt
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019558    4614 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019682    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls podName:5e2f49f4-2884-4804-bb3a-3ee463eeb4f8 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.019672072 +0000 UTC m=+4.852330132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ntl2c" (UID: "5e2f49f4-2884-4804-bb3a-3ee463eeb4f8") : secret "samples-operator-tls" not found
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019699    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert podName:e5393080-e20a-47fc-8dc7-125f42457736 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.019690653 +0000 UTC m=+4.852348712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qrb6h" (UID: "e5393080-e20a-47fc-8dc7-125f42457736") : secret "networking-console-plugin-cert" not found
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019722    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls podName:502f9765-e116-4359-b513-c1d00c90d7de nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.019714009 +0000 UTC m=+4.852372069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls") pod "dns-default-zgjvs" (UID: "502f9765-e116-4359-b513-c1d00c90d7de") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:22.019764    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:22.019786    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019872    4614 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019883    4614 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019894    4614 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79578858d5-6w82j: secret "image-registry-tls" not found
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019913    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls podName:b36bfac6-a600-4029-a446-7de82575fb0a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.019902607 +0000 UTC m=+4.852560677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nbfgg" (UID: "b36bfac6-a600-4029-a446-7de82575fb0a") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:22.020102 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.019928    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls podName:3603dee5-a95f-47e4-9d60-381e82b8afa4 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:24.019920215 +0000 UTC m=+4.852578275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls") pod "image-registry-79578858d5-6w82j" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4") : secret "image-registry-tls" not found
Nov 02 09:35:22.374947 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:22.374863    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/2.log"
Nov 02 09:35:22.963052 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:22.963015    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:22.963486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:22.963468    4614 scope.go:122] "RemoveContainer" containerID="f21e613441812ae795f21576ceb5d8db17b26ad841895c573c1ebb0b76be8f6d"
Nov 02 09:35:22.963712 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:22.963689    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n99gd_openshift-console-operator(2d28887e-4aa9-4209-a51a-fec831136660)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd" podUID="2d28887e-4aa9-4209-a51a-fec831136660"
Nov 02 09:35:22.998246 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:22.998222    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:23.345841 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:23.345814    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zz4q2_92f5b78b-58a0-4d0d-9bf6-97c4943597fd/dns-node-resolver/0.log"
Nov 02 09:35:23.573462 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:23.573427    4614 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:23.573857 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:23.573808    4614 scope.go:122] "RemoveContainer" containerID="f21e613441812ae795f21576ceb5d8db17b26ad841895c573c1ebb0b76be8f6d"
Nov 02 09:35:23.574013 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:23.573994    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n99gd_openshift-console-operator(2d28887e-4aa9-4209-a51a-fec831136660)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd" podUID="2d28887e-4aa9-4209-a51a-fec831136660"
Nov 02 09:35:23.621825 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:23.621750    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:35:24.042005 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:24.041969    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:24.042200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:24.042029    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:24.042200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:24.042050    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:24.042200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:24.042078    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:24.042200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:24.042100    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:24.042200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:24.042115    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:24.042200 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042129    4614 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:24.042200 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042140    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:28.042119866 +0000 UTC m=+8.874777943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : configmap references non-existent config key: service-ca.crt
Nov 02 09:35:24.042200 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042195    4614 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042224    4614 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042230    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls podName:3c01c082-8479-460d-a80d-5e590ab2fcbe nodeName:}" failed. No retries permitted until 2026-11-02 09:35:28.042202763 +0000 UTC m=+8.874860836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mlxbp" (UID: "3c01c082-8479-460d-a80d-5e590ab2fcbe") : secret "cluster-monitoring-operator-tls" not found
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042239    4614 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8fbd47b-96z8h: secret "image-registry-tls" not found
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042237    4614 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042256    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls podName:502f9765-e116-4359-b513-c1d00c90d7de nodeName:}" failed. No retries permitted until 2026-11-02 09:35:28.042243696 +0000 UTC m=+8.874901762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls") pod "dns-default-zgjvs" (UID: "502f9765-e116-4359-b513-c1d00c90d7de") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042195    4614 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042278    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls podName:2daf8a59-8027-42b4-8414-55a0883bc04f nodeName:}" failed. No retries permitted until 2026-11-02 09:35:28.042264951 +0000 UTC m=+8.874923018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls") pod "image-registry-bc8fbd47b-96z8h" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f") : secret "image-registry-tls" not found
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042300    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls podName:5e2f49f4-2884-4804-bb3a-3ee463eeb4f8 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:28.042294098 +0000 UTC m=+8.874952156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ntl2c" (UID: "5e2f49f4-2884-4804-bb3a-3ee463eeb4f8") : secret "samples-operator-tls" not found
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042312    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs podName:961f499a-2ed3-4e0d-b5d2-9194d18b696a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:28.042307856 +0000 UTC m=+8.874965914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs") pod "network-metrics-daemon-v9xng" (UID: "961f499a-2ed3-4e0d-b5d2-9194d18b696a") : secret "metrics-daemon-secret" not found
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:24.042348    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:24.042410    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042428    4614 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:24.042443    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042465    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert podName:e5393080-e20a-47fc-8dc7-125f42457736 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:28.042452547 +0000 UTC m=+8.875110616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qrb6h" (UID: "e5393080-e20a-47fc-8dc7-125f42457736") : secret "networking-console-plugin-cert" not found
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042511    4614 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:24.042528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:24.042516    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:24.043006 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042522    4614 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79578858d5-6w82j: secret "image-registry-tls" not found
Nov 02 09:35:24.043006 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042569    4614 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 09:35:24.043006 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042596    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls podName:3603dee5-a95f-47e4-9d60-381e82b8afa4 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:28.042585305 +0000 UTC m=+8.875243378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls") pod "image-registry-79578858d5-6w82j" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4") : secret "image-registry-tls" not found
Nov 02 09:35:24.043006 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042610    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:28.042602827 +0000 UTC m=+8.875260895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : secret "router-metrics-certs-default" not found
Nov 02 09:35:24.043006 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042547    4614 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:24.043006 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:24.042631    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls podName:b36bfac6-a600-4029-a446-7de82575fb0a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:28.04262641 +0000 UTC m=+8.875284480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nbfgg" (UID: "b36bfac6-a600-4029-a446-7de82575fb0a") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:24.346178 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:24.346107    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xz8m5_77530df7-545e-42b0-ad16-ce934cb10dab/node-ca/0.log"
Nov 02 09:35:24.985196 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:24.985168    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:24.985619 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:24.985451    4614 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 09:35:25.146942 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:25.146917    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xkrcs_26adcc96-ef0b-416f-8b15-e01be59fb5ae/serve-healthcheck-canary/0.log"
Nov 02 09:35:25.747700 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:25.747667    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd_dbd4d998-a5be-4a0a-9113-8f3782df753a/kube-storage-version-migrator-operator/0.log"
Nov 02 09:35:28.081731 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.081694    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:28.081731 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.081736    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.081792    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.081818    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.081846    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.081899    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.081910    4614 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.081928    4614 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.081952    4614 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bc8fbd47b-96z8h: secret "image-registry-tls" not found
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.082001    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs podName:961f499a-2ed3-4e0d-b5d2-9194d18b696a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:36.081980104 +0000 UTC m=+16.914638188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs") pod "network-metrics-daemon-v9xng" (UID: "961f499a-2ed3-4e0d-b5d2-9194d18b696a") : secret "metrics-daemon-secret" not found
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.082023    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls podName:2daf8a59-8027-42b4-8414-55a0883bc04f nodeName:}" failed. No retries permitted until 2026-11-02 09:35:36.082013204 +0000 UTC m=+16.914671265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls") pod "image-registry-bc8fbd47b-96z8h" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f") : secret "image-registry-tls" not found
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.082054    4614 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.081965    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.082119    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls podName:502f9765-e116-4359-b513-c1d00c90d7de nodeName:}" failed. No retries permitted until 2026-11-02 09:35:36.082102488 +0000 UTC m=+16.914760566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls") pod "dns-default-zgjvs" (UID: "502f9765-e116-4359-b513-c1d00c90d7de") : secret "dns-default-metrics-tls" not found
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.082130    4614 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.082145    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.082161    4614 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.082185    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert podName:e5393080-e20a-47fc-8dc7-125f42457736 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:36.082171896 +0000 UTC m=+16.914829980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qrb6h" (UID: "e5393080-e20a-47fc-8dc7-125f42457736") : secret "networking-console-plugin-cert" not found
Nov 02 09:35:28.082210 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.082208    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls podName:b36bfac6-a600-4029-a446-7de82575fb0a nodeName:}" failed. No retries permitted until 2026-11-02 09:35:36.082193912 +0000 UTC m=+16.914851971 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nbfgg" (UID: "b36bfac6-a600-4029-a446-7de82575fb0a") : secret "insights-runtime-extractor-tls" not found
Nov 02 09:35:28.082820 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.082215    4614 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 09:35:28.082820 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.082231    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:28.082820 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.082261    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:28.082820 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.082209    4614 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 09:35:28.082820 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.082327    4614 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79578858d5-6w82j: secret "image-registry-tls" not found
Nov 02 09:35:28.082820 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.082312    4614 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 09:35:28.082820 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.082391    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls podName:3603dee5-a95f-47e4-9d60-381e82b8afa4 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:36.082378953 +0000 UTC m=+16.915037028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls") pod "image-registry-79578858d5-6w82j" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4") : secret "image-registry-tls" not found
Nov 02 09:35:28.082820 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.082407    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:36.082399885 +0000 UTC m=+16.915057943 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : configmap references non-existent config key: service-ca.crt
Nov 02 09:35:28.082820 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:28.082428    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs podName:80746de3-f9f3-4efe-a963-9b37f290fcb7 nodeName:}" failed. No retries permitted until 2026-11-02 09:35:36.08241817 +0000 UTC m=+16.915076235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs") pod "router-default-68d88c448f-z5k5p" (UID: "80746de3-f9f3-4efe-a963-9b37f290fcb7") : secret "router-metrics-certs-default" not found
Nov 02 09:35:28.085885 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.085862    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c01c082-8479-460d-a80d-5e590ab2fcbe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mlxbp\" (UID: \"3c01c082-8479-460d-a80d-5e590ab2fcbe\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:28.085987 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.085897    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e2f49f4-2884-4804-bb3a-3ee463eeb4f8-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ntl2c\" (UID: \"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:28.092221 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.092193    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"
Nov 02 09:35:28.093323 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.093304    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"
Nov 02 09:35:28.370423 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.370344    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp"]
Nov 02 09:35:28.373162 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.373112    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c"]
Nov 02 09:35:28.376143 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.376127    4614 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 09:35:28.394097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:28.393968    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp" event={"ID":"3c01c082-8479-460d-a80d-5e590ab2fcbe","Type":"ContainerStarted","Data":"5a8560ee815d720a654e33c1b39fb6d1ba941cf982ad94672c6a62e633488f8a"}
Nov 02 09:35:29.398316 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:29.398271    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c" event={"ID":"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8","Type":"ContainerStarted","Data":"a8bce5afdd8e94eb3fb534704e2554a798c832d38440cb6be2956aa838423f79"}
Nov 02 09:35:29.999452 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:29.999421    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtrqd"
Nov 02 09:35:30.403528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:30.403469    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mlxbp" event={"ID":"3c01c082-8479-460d-a80d-5e590ab2fcbe","Type":"ContainerStarted","Data":"55229994e034d9a19e640bb7f439363cef3633e99daa6e32ace083e27ea07ddf"}
Nov 02 09:35:31.408699 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:31.408660    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c" event={"ID":"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8","Type":"ContainerStarted","Data":"7df806b6ad7a74042a735a7aefa5f3f09c27a78fee076c97765b04e42e3577da"}
Nov 02 09:35:31.409197 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:31.408713    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ntl2c" event={"ID":"5e2f49f4-2884-4804-bb3a-3ee463eeb4f8","Type":"ContainerStarted","Data":"fff25fa308ae937bfb1c459162bc37dbb3db5e72e22a54b90371ef47f193e3c2"}
Nov 02 09:35:33.309606 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:33.309570    4614 scope.go:122] "RemoveContainer" containerID="f21e613441812ae795f21576ceb5d8db17b26ad841895c573c1ebb0b76be8f6d"
Nov 02 09:35:33.415870 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:33.415849    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 09:35:33.416260 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:33.416244    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/2.log"
Nov 02 09:35:33.416324 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:33.416275    4614 generic.go:356] "Generic (PLEG): container finished" podID="2d28887e-4aa9-4209-a51a-fec831136660" containerID="e64c65e4d94f33e8efaa0da81b6952d0be9d40db507779d875b7b994d25b78fd" exitCode=255
Nov 02 09:35:33.416377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:33.416339    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd" event={"ID":"2d28887e-4aa9-4209-a51a-fec831136660","Type":"ContainerDied","Data":"e64c65e4d94f33e8efaa0da81b6952d0be9d40db507779d875b7b994d25b78fd"}
Nov 02 09:35:33.416417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:33.416401    4614 scope.go:122] "RemoveContainer" containerID="f21e613441812ae795f21576ceb5d8db17b26ad841895c573c1ebb0b76be8f6d"
Nov 02 09:35:33.416692 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:33.416675    4614 scope.go:122] "RemoveContainer" containerID="e64c65e4d94f33e8efaa0da81b6952d0be9d40db507779d875b7b994d25b78fd"
Nov 02 09:35:33.416887 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:33.416868    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n99gd_openshift-console-operator(2d28887e-4aa9-4209-a51a-fec831136660)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd" podUID="2d28887e-4aa9-4209-a51a-fec831136660"
Nov 02 09:35:33.573991 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:33.573917    4614 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:34.420422 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:34.420393    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 09:35:34.420790 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:34.420683    4614 scope.go:122] "RemoveContainer" containerID="e64c65e4d94f33e8efaa0da81b6952d0be9d40db507779d875b7b994d25b78fd"
Nov 02 09:35:34.420869 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:34.420850    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n99gd_openshift-console-operator(2d28887e-4aa9-4209-a51a-fec831136660)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd" podUID="2d28887e-4aa9-4209-a51a-fec831136660"
Nov 02 09:35:36.162656 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.162614    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:36.163163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.162662    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:36.163163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.162688    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:36.163163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.162721    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:36.163163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.162757    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:36.163163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.162917    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:36.163163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.162976    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:36.163163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.163005    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:36.163877 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.163850    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80746de3-f9f3-4efe-a963-9b37f290fcb7-service-ca-bundle\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:36.165411 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.165380    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961f499a-2ed3-4e0d-b5d2-9194d18b696a-metrics-certs\") pod \"network-metrics-daemon-v9xng\" (UID: \"961f499a-2ed3-4e0d-b5d2-9194d18b696a\") " pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:36.165851 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.165823    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502f9765-e116-4359-b513-c1d00c90d7de-metrics-tls\") pod \"dns-default-zgjvs\" (UID: \"502f9765-e116-4359-b513-c1d00c90d7de\") " pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:36.165924 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.165849    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"image-registry-79578858d5-6w82j\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") " pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:36.165924 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.165833    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b36bfac6-a600-4029-a446-7de82575fb0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nbfgg\" (UID: \"b36bfac6-a600-4029-a446-7de82575fb0a\") " pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:36.165924 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.165905    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80746de3-f9f3-4efe-a963-9b37f290fcb7-metrics-certs\") pod \"router-default-68d88c448f-z5k5p\" (UID: \"80746de3-f9f3-4efe-a963-9b37f290fcb7\") " pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:36.166020 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.165929    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"image-registry-bc8fbd47b-96z8h\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") " pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:36.166020 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.165906    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e5393080-e20a-47fc-8dc7-125f42457736-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qrb6h\" (UID: \"e5393080-e20a-47fc-8dc7-125f42457736\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:36.189247 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.189226    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:36.193784 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.193765    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:36.194870 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.194852    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"
Nov 02 09:35:36.197510 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.197491    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v9xng"
Nov 02 09:35:36.201382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.201219    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:36.202315 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.202295    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:36.204022 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.204000    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nbfgg"
Nov 02 09:35:36.484615 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.484585    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68d88c448f-z5k5p"]
Nov 02 09:35:36.487791 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:36.487758    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80746de3_f9f3_4efe_a963_9b37f290fcb7.slice/crio-5d3f0bfb290c5d0d5dcd2ac3aecd20cd05e76f5d40d3d5b804ece8df1af9fbbd WatchSource:0}: Error finding container 5d3f0bfb290c5d0d5dcd2ac3aecd20cd05e76f5d40d3d5b804ece8df1af9fbbd: Status 404 returned error can't find the container with id 5d3f0bfb290c5d0d5dcd2ac3aecd20cd05e76f5d40d3d5b804ece8df1af9fbbd
Nov 02 09:35:36.488540 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.488517    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79578858d5-6w82j"]
Nov 02 09:35:36.491689 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.491666    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qrb6h"]
Nov 02 09:35:36.492630 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:36.492522    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3603dee5_a95f_47e4_9d60_381e82b8afa4.slice/crio-1683a1975a7be12dc656a125c1f98d89e1b711bdd1356a7896682ae05e821fc9 WatchSource:0}: Error finding container 1683a1975a7be12dc656a125c1f98d89e1b711bdd1356a7896682ae05e821fc9: Status 404 returned error can't find the container with id 1683a1975a7be12dc656a125c1f98d89e1b711bdd1356a7896682ae05e821fc9
Nov 02 09:35:36.497467 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:36.497444    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5393080_e20a_47fc_8dc7_125f42457736.slice/crio-fab260d145ee15c2b1f7a4b92c7a4c9e98d5b979209ddbe7c1594ae65e6e7c41 WatchSource:0}: Error finding container fab260d145ee15c2b1f7a4b92c7a4c9e98d5b979209ddbe7c1594ae65e6e7c41: Status 404 returned error can't find the container with id fab260d145ee15c2b1f7a4b92c7a4c9e98d5b979209ddbe7c1594ae65e6e7c41
Nov 02 09:35:36.515065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.515028    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nbfgg"]
Nov 02 09:35:36.552176 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.552152    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v9xng"]
Nov 02 09:35:36.555171 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.555148    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zgjvs"]
Nov 02 09:35:36.559600 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:36.559565    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bc8fbd47b-96z8h"]
Nov 02 09:35:36.572570 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:36.572536    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2daf8a59_8027_42b4_8414_55a0883bc04f.slice/crio-7b0e6d8652d2c9740ae4c7d7f9632e19980b45415161dbdb98df62cbaddf935d WatchSource:0}: Error finding container 7b0e6d8652d2c9740ae4c7d7f9632e19980b45415161dbdb98df62cbaddf935d: Status 404 returned error can't find the container with id 7b0e6d8652d2c9740ae4c7d7f9632e19980b45415161dbdb98df62cbaddf935d
Nov 02 09:35:37.436466 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:37.436428    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nbfgg" event={"ID":"b36bfac6-a600-4029-a446-7de82575fb0a","Type":"ContainerStarted","Data":"6a027748c21dbfd9767a6c8d6e42cfb126bf6e0518c016ebd166af01138d65d0"}
Nov 02 09:35:37.436852 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:37.436503    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nbfgg" event={"ID":"b36bfac6-a600-4029-a446-7de82575fb0a","Type":"ContainerStarted","Data":"b11dc5a14e9142c535d5a3b599e660731120794cd9153850647c897506807d1b"}
Nov 02 09:35:37.438563 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:37.438532    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h" event={"ID":"2daf8a59-8027-42b4-8414-55a0883bc04f","Type":"ContainerStarted","Data":"9ef673a13457a67874acf76fc7abf196ca07b9dba3da2855626c756431fc4071"}
Nov 02 09:35:37.438651 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:37.438592    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h" event={"ID":"2daf8a59-8027-42b4-8414-55a0883bc04f","Type":"ContainerStarted","Data":"7b0e6d8652d2c9740ae4c7d7f9632e19980b45415161dbdb98df62cbaddf935d"}
Nov 02 09:35:37.439438 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:37.439414    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:37.441038 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:37.440977    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v9xng" event={"ID":"961f499a-2ed3-4e0d-b5d2-9194d18b696a","Type":"ContainerStarted","Data":"2fe0c7468a196c1c58c12b07d8e1b5d55dda2e7153ce557bca23be7aee717eb1"}
Nov 02 09:35:37.444677 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:37.443728    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79578858d5-6w82j" event={"ID":"3603dee5-a95f-47e4-9d60-381e82b8afa4","Type":"ContainerStarted","Data":"ce5c937774fd7879c454b046316f594d891d0286f466b45f23fc7661c1704606"}
Nov 02 09:35:37.444677 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:37.443770    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79578858d5-6w82j" event={"ID":"3603dee5-a95f-47e4-9d60-381e82b8afa4","Type":"ContainerStarted","Data":"1683a1975a7be12dc656a125c1f98d89e1b711bdd1356a7896682ae05e821fc9"}
Nov 02 09:35:37.444677 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:37.444211    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:35:37.447289 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:37.447266    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68d88c448f-z5k5p" event={"ID":"80746de3-f9f3-4efe-a963-9b37f290fcb7","Type":"ContainerStarted","Data":"05f300f08208580755784042c78cde77d6e5a6a6937d76f4408f3f7880e768d6"}
Nov 02 09:35:37.447412 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:37.447305    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68d88c448f-z5k5p" event={"ID":"80746de3-f9f3-4efe-a963-9b37f290fcb7","Type":"ContainerStarted","Data":"5d3f0bfb290c5d0d5dcd2ac3aecd20cd05e76f5d40d3d5b804ece8df1af9fbbd"}
Nov 02 09:35:37.449560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:37.449534    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h" event={"ID":"e5393080-e20a-47fc-8dc7-125f42457736","Type":"ContainerStarted","Data":"fab260d145ee15c2b1f7a4b92c7a4c9e98d5b979209ddbe7c1594ae65e6e7c41"}
Nov 02 09:35:37.450620 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:37.450597    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgjvs" event={"ID":"502f9765-e116-4359-b513-c1d00c90d7de","Type":"ContainerStarted","Data":"4240115d2617361bae7e2f1ee550122f6968527a87fe00feb396d32daf10780c"}
Nov 02 09:35:37.728075 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:37.727516    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79578858d5-6w82j"]
Nov 02 09:35:38.189582 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:38.189548    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:38.192324 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:38.192301    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:38.456474 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:38.456386    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nbfgg" event={"ID":"b36bfac6-a600-4029-a446-7de82575fb0a","Type":"ContainerStarted","Data":"9109ac692f4525a3eadb68e2211352caa4ed840f1fb92916059ffdc2128c4693"}
Nov 02 09:35:38.458387 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:38.458343    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v9xng" event={"ID":"961f499a-2ed3-4e0d-b5d2-9194d18b696a","Type":"ContainerStarted","Data":"269c92bedeb6cca8c630a6718360b95b27dc4b10cce3749d22ce2e700895f88a"}
Nov 02 09:35:38.458530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:38.458412    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v9xng" event={"ID":"961f499a-2ed3-4e0d-b5d2-9194d18b696a","Type":"ContainerStarted","Data":"673b7b023a4bea3b7c150f1fc7ed7c8755b0f1714cc95f2ae055a1b2067a23bc"}
Nov 02 09:35:38.459909 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:38.459880    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-qrb6h" event={"ID":"e5393080-e20a-47fc-8dc7-125f42457736","Type":"ContainerStarted","Data":"a1122b6006ebc28a10cea84f501dbdd59d336f8fc2edb720b7b9e6a48c73e244"}
Nov 02 09:35:38.460585 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:38.460564    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:38.461673 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:38.461656    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-68d88c448f-z5k5p"
Nov 02 09:35:39.465571 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:39.465474    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgjvs" event={"ID":"502f9765-e116-4359-b513-c1d00c90d7de","Type":"ContainerStarted","Data":"19046cd3b6e253095fb1d3e42a2cce7007414978057ad57a7eb0593e171ba58b"}
Nov 02 09:35:39.465571 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:39.465550    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgjvs" event={"ID":"502f9765-e116-4359-b513-c1d00c90d7de","Type":"ContainerStarted","Data":"91909d279978f188866f688e5e9912ffeb84eef94683790dbad3ce0c10b44f9c"}
Nov 02 09:35:39.466008 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:39.465710    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:40.471665 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:40.471630    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nbfgg" event={"ID":"b36bfac6-a600-4029-a446-7de82575fb0a","Type":"ContainerStarted","Data":"c8c7d7aba1057fa1bf0e8e3caf62014b9b6020dda9fd8e9c4478edce5bc22d99"}
Nov 02 09:35:42.963363 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:42.963314    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:42.963723 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:42.963714    4614 scope.go:122] "RemoveContainer" containerID="e64c65e4d94f33e8efaa0da81b6952d0be9d40db507779d875b7b994d25b78fd"
Nov 02 09:35:42.963902 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:35:42.963883    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n99gd_openshift-console-operator(2d28887e-4aa9-4209-a51a-fec831136660)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd" podUID="2d28887e-4aa9-4209-a51a-fec831136660"
Nov 02 09:35:44.266822 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.266789    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nd59c"]
Nov 02 09:35:44.272321 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.272301    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.341174 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.341119    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 09:35:44.341338 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.341205    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-phbth\""
Nov 02 09:35:44.341338 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.341207    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 09:35:44.361918 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.361893    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 09:35:44.362052 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.361966    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 09:35:44.431505 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.431475    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-wtmp\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.431643 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.431516    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-root\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.431643 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.431542    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-sys\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.431734 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.431659    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tdwq\" (UniqueName: \"kubernetes.io/projected/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-kube-api-access-4tdwq\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.431734 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.431688    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-tls\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.431734 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.431718    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-accelerators-collector-config\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.431848 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.431797    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-metrics-client-ca\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.431848 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.431822    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-textfile\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.431848 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.431844    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.532458 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.532378    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-accelerators-collector-config\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.532458 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.532420    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-metrics-client-ca\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.532458 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.532455    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-textfile\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.532714 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.532481    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.532714 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.532533    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-wtmp\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.532714 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.532676    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-wtmp\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.532861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.532718    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-root\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.532861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.532738    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-root\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.532861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.532753    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-sys\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.532861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.532786    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tdwq\" (UniqueName: \"kubernetes.io/projected/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-kube-api-access-4tdwq\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.532861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.532807    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-tls\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.532861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.532849    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-sys\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.533110 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.532891    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-textfile\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.533110 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.533073    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-metrics-client-ca\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.533179 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.533149    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-accelerators-collector-config\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.535108 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.535082    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-tls\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.535213 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.535082    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.544837 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.544810    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tdwq\" (UniqueName: \"kubernetes.io/projected/44a54ffa-79c6-4dfb-9acc-f045c5538d1f-kube-api-access-4tdwq\") pod \"node-exporter-nd59c\" (UID: \"44a54ffa-79c6-4dfb-9acc-f045c5538d1f\") " pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.589394 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.589370    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nd59c"
Nov 02 09:35:44.604761 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:35:44.604735    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a54ffa_79c6_4dfb_9acc_f045c5538d1f.slice/crio-4c368306277ae3e032e168bb8682901dce60fd81f4f9609bf4b122f2e696b609 WatchSource:0}: Error finding container 4c368306277ae3e032e168bb8682901dce60fd81f4f9609bf4b122f2e696b609: Status 404 returned error can't find the container with id 4c368306277ae3e032e168bb8682901dce60fd81f4f9609bf4b122f2e696b609
Nov 02 09:35:44.955987 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:44.955905    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xzrll"
Nov 02 09:35:45.491777 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:45.491724    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nd59c" event={"ID":"44a54ffa-79c6-4dfb-9acc-f045c5538d1f","Type":"ContainerStarted","Data":"4c368306277ae3e032e168bb8682901dce60fd81f4f9609bf4b122f2e696b609"}
Nov 02 09:35:46.497492 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:46.497460    4614 generic.go:356] "Generic (PLEG): container finished" podID="44a54ffa-79c6-4dfb-9acc-f045c5538d1f" containerID="ab69c493418fea0bea6625b548534c11730879164952d73f5033cafccaaa3069" exitCode=0
Nov 02 09:35:46.497935 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:46.497553    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nd59c" event={"ID":"44a54ffa-79c6-4dfb-9acc-f045c5538d1f","Type":"ContainerDied","Data":"ab69c493418fea0bea6625b548534c11730879164952d73f5033cafccaaa3069"}
Nov 02 09:35:47.503551 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:47.503507    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nd59c" event={"ID":"44a54ffa-79c6-4dfb-9acc-f045c5538d1f","Type":"ContainerStarted","Data":"ee3e14419a5e26d57728797bde7418904d49d4d931455b6c26392a1172033817"}
Nov 02 09:35:47.504024 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:47.503574    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nd59c" event={"ID":"44a54ffa-79c6-4dfb-9acc-f045c5538d1f","Type":"ContainerStarted","Data":"4aef74524629eecee5770a37a6366fd76e538ee2b0caf65c0cb58395a764d238"}
Nov 02 09:35:47.527318 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:47.527263    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nd59c" podStartSLOduration=2.36282103 podStartE2EDuration="3.527251139s" totalImagesPullingTime="1.098603306s" totalInitContainerRuntime="65.826803ms" isStatefulPod=true podCreationTimestamp="2026-11-02 09:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:35:47.525757604 +0000 UTC m=+28.358415710" watchObservedRunningTime="2026-11-02 09:35:47.527251139 +0000 UTC m=+28.359909219"
Nov 02 09:35:49.474336 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:49.474311    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zgjvs"
Nov 02 09:35:53.625046 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:53.625017    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h5bzx"
Nov 02 09:35:56.206157 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:56.206122    4614 patch_prober.go:29] interesting pod/image-registry-bc8fbd47b-96z8h container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 09:35:56.206627 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:56.206175    4614 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h" podUID="2daf8a59-8027-42b4-8414-55a0883bc04f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 09:35:56.308815 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:56.308780    4614 scope.go:122] "RemoveContainer" containerID="e64c65e4d94f33e8efaa0da81b6952d0be9d40db507779d875b7b994d25b78fd"
Nov 02 09:35:56.541406 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:56.541378    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 09:35:56.541578 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:56.541460    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd" event={"ID":"2d28887e-4aa9-4209-a51a-fec831136660","Type":"ContainerStarted","Data":"c9b70ca20af23db615d06e9a4ef9a9f2878a9ff36df42daf891d142e970f3f9e"}
Nov 02 09:35:56.541764 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:56.541743    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:56.660588 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:56.660558    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-n99gd"
Nov 02 09:35:59.470600 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:59.470571    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:35:59.471752 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:35:59.471731    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:36:00.358936 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:00.358819    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bc8fbd47b-96z8h"]
Nov 02 09:36:04.485780 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.485734    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79578858d5-6w82j" podUID="3603dee5-a95f-47e4-9d60-381e82b8afa4" containerName="registry" containerID="cri-o://ce5c937774fd7879c454b046316f594d891d0286f466b45f23fc7661c1704606" gracePeriod=30
Nov 02 09:36:04.725905 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.725879    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:36:04.784765 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.784680    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-kube-api-access-xhsmx\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-kube-api-access-xhsmx\") pod \"3603dee5-a95f-47e4-9d60-381e82b8afa4\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") "
Nov 02 09:36:04.784765 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.784726    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-certificates\") pod \"3603dee5-a95f-47e4-9d60-381e82b8afa4\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") "
Nov 02 09:36:04.784765 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.784754    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-image-registry-private-configuration\") pod \"3603dee5-a95f-47e4-9d60-381e82b8afa4\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") "
Nov 02 09:36:04.785084 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.784806    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3603dee5-a95f-47e4-9d60-381e82b8afa4-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3603dee5-a95f-47e4-9d60-381e82b8afa4-ca-trust-extracted\") pod \"3603dee5-a95f-47e4-9d60-381e82b8afa4\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") "
Nov 02 09:36:04.785084 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.784822    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-bound-sa-token\") pod \"3603dee5-a95f-47e4-9d60-381e82b8afa4\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") "
Nov 02 09:36:04.785084 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.784844    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-installation-pull-secrets\") pod \"3603dee5-a95f-47e4-9d60-381e82b8afa4\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") "
Nov 02 09:36:04.785084 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.784872    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") pod \"3603dee5-a95f-47e4-9d60-381e82b8afa4\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") "
Nov 02 09:36:04.785084 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.784927    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-trusted-ca\") pod \"3603dee5-a95f-47e4-9d60-381e82b8afa4\" (UID: \"3603dee5-a95f-47e4-9d60-381e82b8afa4\") "
Nov 02 09:36:04.785337 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.785206    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-certificates" pod "3603dee5-a95f-47e4-9d60-381e82b8afa4" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:36:04.785542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.785521    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-trusted-ca" pod "3603dee5-a95f-47e4-9d60-381e82b8afa4" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:36:04.787751 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.787724    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-bound-sa-token" pod "3603dee5-a95f-47e4-9d60-381e82b8afa4" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:36:04.787871 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.787779    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-image-registry-private-configuration" pod "3603dee5-a95f-47e4-9d60-381e82b8afa4" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:36:04.787871 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.787852    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-installation-pull-secrets" pod "3603dee5-a95f-47e4-9d60-381e82b8afa4" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:36:04.787871 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.787850    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-kube-api-access-xhsmx" pod "3603dee5-a95f-47e4-9d60-381e82b8afa4" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4"). InnerVolumeSpecName "kube-api-access-xhsmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:36:04.788062 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.788039    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls" pod "3603dee5-a95f-47e4-9d60-381e82b8afa4" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:36:04.794233 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.794202    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3603dee5-a95f-47e4-9d60-381e82b8afa4-ca-trust-extracted" pod "3603dee5-a95f-47e4-9d60-381e82b8afa4" (UID: "3603dee5-a95f-47e4-9d60-381e82b8afa4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:36:04.885932 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.885895    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhsmx\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-kube-api-access-xhsmx\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:04.885932 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.885925    4614 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-certificates\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:04.885932 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.885935    4614 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-image-registry-private-configuration\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:04.886154 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.885946    4614 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3603dee5-a95f-47e4-9d60-381e82b8afa4-ca-trust-extracted\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:04.886154 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.885957    4614 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-bound-sa-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:04.886154 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.885966    4614 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3603dee5-a95f-47e4-9d60-381e82b8afa4-installation-pull-secrets\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:04.886154 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.885975    4614 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603dee5-a95f-47e4-9d60-381e82b8afa4-registry-tls\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:04.886154 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:04.885984    4614 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3603dee5-a95f-47e4-9d60-381e82b8afa4-trusted-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:05.577486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:05.577403    4614 generic.go:356] "Generic (PLEG): container finished" podID="3603dee5-a95f-47e4-9d60-381e82b8afa4" containerID="ce5c937774fd7879c454b046316f594d891d0286f466b45f23fc7661c1704606" exitCode=0
Nov 02 09:36:05.577486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:05.577455    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79578858d5-6w82j"
Nov 02 09:36:05.577486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:05.577472    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79578858d5-6w82j" event={"ID":"3603dee5-a95f-47e4-9d60-381e82b8afa4","Type":"ContainerDied","Data":"ce5c937774fd7879c454b046316f594d891d0286f466b45f23fc7661c1704606"}
Nov 02 09:36:05.578011 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:05.577525    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79578858d5-6w82j" event={"ID":"3603dee5-a95f-47e4-9d60-381e82b8afa4","Type":"ContainerDied","Data":"1683a1975a7be12dc656a125c1f98d89e1b711bdd1356a7896682ae05e821fc9"}
Nov 02 09:36:05.578011 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:05.577545    4614 scope.go:122] "RemoveContainer" containerID="ce5c937774fd7879c454b046316f594d891d0286f466b45f23fc7661c1704606"
Nov 02 09:36:05.585938 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:05.585920    4614 scope.go:122] "RemoveContainer" containerID="ce5c937774fd7879c454b046316f594d891d0286f466b45f23fc7661c1704606"
Nov 02 09:36:05.586216 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:36:05.586191    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce5c937774fd7879c454b046316f594d891d0286f466b45f23fc7661c1704606\": container with ID starting with ce5c937774fd7879c454b046316f594d891d0286f466b45f23fc7661c1704606 not found: ID does not exist" containerID="ce5c937774fd7879c454b046316f594d891d0286f466b45f23fc7661c1704606"
Nov 02 09:36:05.586314 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:05.586225    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5c937774fd7879c454b046316f594d891d0286f466b45f23fc7661c1704606"} err="failed to get container status \"ce5c937774fd7879c454b046316f594d891d0286f466b45f23fc7661c1704606\": rpc error: code = NotFound desc = could not find container \"ce5c937774fd7879c454b046316f594d891d0286f466b45f23fc7661c1704606\": container with ID starting with ce5c937774fd7879c454b046316f594d891d0286f466b45f23fc7661c1704606 not found: ID does not exist"
Nov 02 09:36:05.596623 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:05.596599    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79578858d5-6w82j"]
Nov 02 09:36:05.601040 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:05.601018    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79578858d5-6w82j"]
Nov 02 09:36:07.313169 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:07.313137    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3603dee5-a95f-47e4-9d60-381e82b8afa4" path="/var/lib/kubelet/pods/3603dee5-a95f-47e4-9d60-381e82b8afa4/volumes"
Nov 02 09:36:09.634857 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:36:09.634821    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:36:25.378714 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.378670    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h" podUID="2daf8a59-8027-42b4-8414-55a0883bc04f" containerName="registry" containerID="cri-o://9ef673a13457a67874acf76fc7abf196ca07b9dba3da2855626c756431fc4071" gracePeriod=30
Nov 02 09:36:25.619926 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.619895    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:36:25.654432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.654346    4614 generic.go:356] "Generic (PLEG): container finished" podID="2daf8a59-8027-42b4-8414-55a0883bc04f" containerID="9ef673a13457a67874acf76fc7abf196ca07b9dba3da2855626c756431fc4071" exitCode=0
Nov 02 09:36:25.654432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.654412    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h"
Nov 02 09:36:25.654573 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.654458    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h" event={"ID":"2daf8a59-8027-42b4-8414-55a0883bc04f","Type":"ContainerDied","Data":"9ef673a13457a67874acf76fc7abf196ca07b9dba3da2855626c756431fc4071"}
Nov 02 09:36:25.654573 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.654516    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bc8fbd47b-96z8h" event={"ID":"2daf8a59-8027-42b4-8414-55a0883bc04f","Type":"ContainerDied","Data":"7b0e6d8652d2c9740ae4c7d7f9632e19980b45415161dbdb98df62cbaddf935d"}
Nov 02 09:36:25.654573 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.654547    4614 scope.go:122] "RemoveContainer" containerID="9ef673a13457a67874acf76fc7abf196ca07b9dba3da2855626c756431fc4071"
Nov 02 09:36:25.662262 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.662242    4614 scope.go:122] "RemoveContainer" containerID="9ef673a13457a67874acf76fc7abf196ca07b9dba3da2855626c756431fc4071"
Nov 02 09:36:25.662522 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:36:25.662505    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef673a13457a67874acf76fc7abf196ca07b9dba3da2855626c756431fc4071\": container with ID starting with 9ef673a13457a67874acf76fc7abf196ca07b9dba3da2855626c756431fc4071 not found: ID does not exist" containerID="9ef673a13457a67874acf76fc7abf196ca07b9dba3da2855626c756431fc4071"
Nov 02 09:36:25.662620 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.662531    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef673a13457a67874acf76fc7abf196ca07b9dba3da2855626c756431fc4071"} err="failed to get container status \"9ef673a13457a67874acf76fc7abf196ca07b9dba3da2855626c756431fc4071\": rpc error: code = NotFound desc = could not find container \"9ef673a13457a67874acf76fc7abf196ca07b9dba3da2855626c756431fc4071\": container with ID starting with 9ef673a13457a67874acf76fc7abf196ca07b9dba3da2855626c756431fc4071 not found: ID does not exist"
Nov 02 09:36:25.754429 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.754400    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-certificates\") pod \"2daf8a59-8027-42b4-8414-55a0883bc04f\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") "
Nov 02 09:36:25.754564 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.754442    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-image-registry-private-configuration\") pod \"2daf8a59-8027-42b4-8414-55a0883bc04f\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") "
Nov 02 09:36:25.754564 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.754491    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-bound-sa-token\") pod \"2daf8a59-8027-42b4-8414-55a0883bc04f\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") "
Nov 02 09:36:25.754564 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.754510    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") pod \"2daf8a59-8027-42b4-8414-55a0883bc04f\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") "
Nov 02 09:36:25.754564 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.754529    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-trusted-ca\") pod \"2daf8a59-8027-42b4-8414-55a0883bc04f\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") "
Nov 02 09:36:25.754564 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.754547    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2daf8a59-8027-42b4-8414-55a0883bc04f-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2daf8a59-8027-42b4-8414-55a0883bc04f-ca-trust-extracted\") pod \"2daf8a59-8027-42b4-8414-55a0883bc04f\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") "
Nov 02 09:36:25.754761 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.754585    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-installation-pull-secrets\") pod \"2daf8a59-8027-42b4-8414-55a0883bc04f\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") "
Nov 02 09:36:25.754761 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.754627    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-kube-api-access-v5ql9\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-kube-api-access-v5ql9\") pod \"2daf8a59-8027-42b4-8414-55a0883bc04f\" (UID: \"2daf8a59-8027-42b4-8414-55a0883bc04f\") "
Nov 02 09:36:25.755002 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.754953    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-certificates" pod "2daf8a59-8027-42b4-8414-55a0883bc04f" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:36:25.755120 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.755101    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-trusted-ca" pod "2daf8a59-8027-42b4-8414-55a0883bc04f" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:36:25.757015 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.756984    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-image-registry-private-configuration" pod "2daf8a59-8027-42b4-8414-55a0883bc04f" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:36:25.757121 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.757080    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-installation-pull-secrets" pod "2daf8a59-8027-42b4-8414-55a0883bc04f" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:36:25.757186 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.757167    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-kube-api-access-v5ql9" pod "2daf8a59-8027-42b4-8414-55a0883bc04f" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f"). InnerVolumeSpecName "kube-api-access-v5ql9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:36:25.757246 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.757201    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-bound-sa-token" pod "2daf8a59-8027-42b4-8414-55a0883bc04f" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:36:25.757290 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.757238    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls" pod "2daf8a59-8027-42b4-8414-55a0883bc04f" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:36:25.761505 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.761482    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2daf8a59-8027-42b4-8414-55a0883bc04f-ca-trust-extracted" pod "2daf8a59-8027-42b4-8414-55a0883bc04f" (UID: "2daf8a59-8027-42b4-8414-55a0883bc04f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:36:25.856074 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.856043    4614 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-bound-sa-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:25.856074 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.856069    4614 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-tls\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:25.856074 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.856079    4614 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-trusted-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:25.856291 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.856089    4614 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2daf8a59-8027-42b4-8414-55a0883bc04f-ca-trust-extracted\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:25.856291 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.856098    4614 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-installation-pull-secrets\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:25.856291 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.856109    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5ql9\" (UniqueName: \"kubernetes.io/projected/2daf8a59-8027-42b4-8414-55a0883bc04f-kube-api-access-v5ql9\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:25.856291 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.856119    4614 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2daf8a59-8027-42b4-8414-55a0883bc04f-registry-certificates\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:25.856291 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.856128    4614 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2daf8a59-8027-42b4-8414-55a0883bc04f-image-registry-private-configuration\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:36:25.975596 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.975562    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bc8fbd47b-96z8h"]
Nov 02 09:36:25.979440 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:25.979421    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bc8fbd47b-96z8h"]
Nov 02 09:36:26.659580 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:26.659505    4614 generic.go:356] "Generic (PLEG): container finished" podID="dbd4d998-a5be-4a0a-9113-8f3782df753a" containerID="f0b11800a2ea74eb653c2b80deda00c5a253da6dfb8f06203a70d562d17991b6" exitCode=0
Nov 02 09:36:26.659932 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:26.659578    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd" event={"ID":"dbd4d998-a5be-4a0a-9113-8f3782df753a","Type":"ContainerDied","Data":"f0b11800a2ea74eb653c2b80deda00c5a253da6dfb8f06203a70d562d17991b6"}
Nov 02 09:36:26.659932 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:26.659869    4614 scope.go:122] "RemoveContainer" containerID="f0b11800a2ea74eb653c2b80deda00c5a253da6dfb8f06203a70d562d17991b6"
Nov 02 09:36:27.316576 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:27.313322    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2daf8a59-8027-42b4-8414-55a0883bc04f" path="/var/lib/kubelet/pods/2daf8a59-8027-42b4-8414-55a0883bc04f/volumes"
Nov 02 09:36:27.664321 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:27.664240    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6hwvd" event={"ID":"dbd4d998-a5be-4a0a-9113-8f3782df753a","Type":"ContainerStarted","Data":"f9cdbc6d1c4383c9d9e2095b31c322a772a84fc731b8d0acac5d43b7af74a76b"}
Nov 02 09:36:39.634038 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:36:39.633998    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:36:41.713298 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:41.713217    4614 generic.go:356] "Generic (PLEG): container finished" podID="a2608084-c9df-4998-a521-fb2239b3c349" containerID="d1547b7165a7f0ccd0408d70640dd252dbe0113b894fe60698dec286042ce221" exitCode=0
Nov 02 09:36:41.713680 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:41.713289    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw" event={"ID":"a2608084-c9df-4998-a521-fb2239b3c349","Type":"ContainerDied","Data":"d1547b7165a7f0ccd0408d70640dd252dbe0113b894fe60698dec286042ce221"}
Nov 02 09:36:41.713680 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:41.713614    4614 scope.go:122] "RemoveContainer" containerID="d1547b7165a7f0ccd0408d70640dd252dbe0113b894fe60698dec286042ce221"
Nov 02 09:36:42.718484 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:42.718446    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cwfdw" event={"ID":"a2608084-c9df-4998-a521-fb2239b3c349","Type":"ContainerStarted","Data":"3524c3f61b5459dfa4762f2922129a89a20faf1c2f57f9a08c767a688a42ff98"}
Nov 02 09:36:51.760537 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:51.760501    4614 generic.go:356] "Generic (PLEG): container finished" podID="85741a00-b702-46cb-a90c-e000bf9976f0" containerID="c6f9f6589c6fd87152a37df3f40c37fcd34c20735131ab3e8ee4064de74a8f07" exitCode=0
Nov 02 09:36:51.761094 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:51.760581    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-pdw8n" event={"ID":"85741a00-b702-46cb-a90c-e000bf9976f0","Type":"ContainerDied","Data":"c6f9f6589c6fd87152a37df3f40c37fcd34c20735131ab3e8ee4064de74a8f07"}
Nov 02 09:36:51.761094 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:51.761004    4614 scope.go:122] "RemoveContainer" containerID="c6f9f6589c6fd87152a37df3f40c37fcd34c20735131ab3e8ee4064de74a8f07"
Nov 02 09:36:52.251817 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.251783    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-79lct-772qs"]
Nov 02 09:36:52.252082 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.252070    4614 cpu_manager.go:414] "containerMap: removing container" podUID="2daf8a59-8027-42b4-8414-55a0883bc04f" containerName="registry"
Nov 02 09:36:52.252143 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.252083    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2daf8a59-8027-42b4-8414-55a0883bc04f" containerName="registry"
Nov 02 09:36:52.252143 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.252094    4614 cpu_manager.go:414] "containerMap: removing container" podUID="3603dee5-a95f-47e4-9d60-381e82b8afa4" containerName="registry"
Nov 02 09:36:52.252143 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.252099    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3603dee5-a95f-47e4-9d60-381e82b8afa4" containerName="registry"
Nov 02 09:36:52.252233 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.252169    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="2daf8a59-8027-42b4-8414-55a0883bc04f" containerName="registry"
Nov 02 09:36:52.252233 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.252178    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="3603dee5-a95f-47e4-9d60-381e82b8afa4" containerName="registry"
Nov 02 09:36:52.255334 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.255312    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:36:52.263185 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.263161    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-79lct-772qs"]
Nov 02 09:36:52.366283 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.366255    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-trusted-ca-bundle\") pod \"periodic-gathering-79lct-772qs\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") " pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:36:52.366446 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.366293    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-service-ca-bundle\") pod \"periodic-gathering-79lct-772qs\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") " pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:36:52.366446 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.366323    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-archives-path\") pod \"periodic-gathering-79lct-772qs\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") " pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:36:52.366446 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.366427    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24l9p\" (UniqueName: \"kubernetes.io/projected/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-kube-api-access-24l9p\") pod \"periodic-gathering-79lct-772qs\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") " pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:36:52.467690 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.467659    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-trusted-ca-bundle\") pod \"periodic-gathering-79lct-772qs\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") " pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:36:52.467841 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.467700    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-service-ca-bundle\") pod \"periodic-gathering-79lct-772qs\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") " pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:36:52.467841 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.467746    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-archives-path\") pod \"periodic-gathering-79lct-772qs\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") " pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:36:52.467841 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.467773    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24l9p\" (UniqueName: \"kubernetes.io/projected/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-kube-api-access-24l9p\") pod \"periodic-gathering-79lct-772qs\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") " pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:36:52.468194 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.468171    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-archives-path\") pod \"periodic-gathering-79lct-772qs\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") " pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:36:52.468434 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.468417    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-service-ca-bundle\") pod \"periodic-gathering-79lct-772qs\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") " pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:36:52.468682 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.468665    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-trusted-ca-bundle\") pod \"periodic-gathering-79lct-772qs\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") " pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:36:52.480137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.480114    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24l9p\" (UniqueName: \"kubernetes.io/projected/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-kube-api-access-24l9p\") pod \"periodic-gathering-79lct-772qs\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") " pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:36:52.566641 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.566572    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:36:52.765730 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.765692    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-pdw8n" event={"ID":"85741a00-b702-46cb-a90c-e000bf9976f0","Type":"ContainerStarted","Data":"c26c45ef413aa25cc4a0325fe4ceb9f78379ee2183e8df4f5af6f29d2d2a5855"}
Nov 02 09:36:52.838974 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:52.838894    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-79lct-772qs"]
Nov 02 09:36:52.842101 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:36:52.842065    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bfa1c8_8148_4bdb_abd6_cc24256fd371.slice/crio-4d8ba2fad85955f6f79f960f3c0ebd40962f89a662d7ecf13fb47773c3c22803 WatchSource:0}: Error finding container 4d8ba2fad85955f6f79f960f3c0ebd40962f89a662d7ecf13fb47773c3c22803: Status 404 returned error can't find the container with id 4d8ba2fad85955f6f79f960f3c0ebd40962f89a662d7ecf13fb47773c3c22803
Nov 02 09:36:53.770564 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:53.770530    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-79lct-772qs" event={"ID":"b0bfa1c8-8148-4bdb-abd6-cc24256fd371","Type":"ContainerStarted","Data":"66ce9d3e43f6c3ae2a03c727aacb4a30e321c6e94a9a5e0ac9f0ed237d87f87b"}
Nov 02 09:36:53.770940 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:53.770579    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-79lct-772qs" event={"ID":"b0bfa1c8-8148-4bdb-abd6-cc24256fd371","Type":"ContainerStarted","Data":"4d8ba2fad85955f6f79f960f3c0ebd40962f89a662d7ecf13fb47773c3c22803"}
Nov 02 09:36:53.787483 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:36:53.787440    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-79lct-772qs" podStartSLOduration=1.787426645 podStartE2EDuration="1.787426645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:36:53.787391923 +0000 UTC m=+94.620050003" watchObservedRunningTime="2026-11-02 09:36:53.787426645 +0000 UTC m=+94.620084717"
Nov 02 09:37:09.634425 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:37:09.634388    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:37:39.635251 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:37:39.635212    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:38:09.635715 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:38:09.635677    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:38:17.081280 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:17.081247    4614 generic.go:356] "Generic (PLEG): container finished" podID="b0bfa1c8-8148-4bdb-abd6-cc24256fd371" containerID="66ce9d3e43f6c3ae2a03c727aacb4a30e321c6e94a9a5e0ac9f0ed237d87f87b" exitCode=0
Nov 02 09:38:17.081713 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:17.081321    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-79lct-772qs" event={"ID":"b0bfa1c8-8148-4bdb-abd6-cc24256fd371","Type":"ContainerDied","Data":"66ce9d3e43f6c3ae2a03c727aacb4a30e321c6e94a9a5e0ac9f0ed237d87f87b"}
Nov 02 09:38:18.209246 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:18.209225    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:38:18.308408 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:18.308372    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-kube-api-access-24l9p\" (UniqueName: \"kubernetes.io/projected/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-kube-api-access-24l9p\") pod \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") "
Nov 02 09:38:18.308554 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:18.308466    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-archives-path\") pod \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") "
Nov 02 09:38:18.308554 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:18.308503    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-service-ca-bundle\") pod \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") "
Nov 02 09:38:18.308554 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:18.308533    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-trusted-ca-bundle\") pod \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\" (UID: \"b0bfa1c8-8148-4bdb-abd6-cc24256fd371\") "
Nov 02 09:38:18.308872 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:18.308843    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-archives-path" pod "b0bfa1c8-8148-4bdb-abd6-cc24256fd371" (UID: "b0bfa1c8-8148-4bdb-abd6-cc24256fd371"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:38:18.308947 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:18.308855    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-service-ca-bundle" pod "b0bfa1c8-8148-4bdb-abd6-cc24256fd371" (UID: "b0bfa1c8-8148-4bdb-abd6-cc24256fd371"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:38:18.308947 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:18.308883    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-trusted-ca-bundle" pod "b0bfa1c8-8148-4bdb-abd6-cc24256fd371" (UID: "b0bfa1c8-8148-4bdb-abd6-cc24256fd371"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:38:18.310605 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:18.310583    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-kube-api-access-24l9p" pod "b0bfa1c8-8148-4bdb-abd6-cc24256fd371" (UID: "b0bfa1c8-8148-4bdb-abd6-cc24256fd371"). InnerVolumeSpecName "kube-api-access-24l9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:38:18.409273 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:18.409198    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24l9p\" (UniqueName: \"kubernetes.io/projected/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-kube-api-access-24l9p\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:38:18.409273 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:18.409228    4614 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-archives-path\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:38:18.409273 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:18.409240    4614 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-service-ca-bundle\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:38:18.409273 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:18.409249    4614 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0bfa1c8-8148-4bdb-abd6-cc24256fd371-trusted-ca-bundle\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:38:19.089412 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:19.089371    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-79lct-772qs" event={"ID":"b0bfa1c8-8148-4bdb-abd6-cc24256fd371","Type":"ContainerDied","Data":"4d8ba2fad85955f6f79f960f3c0ebd40962f89a662d7ecf13fb47773c3c22803"}
Nov 02 09:38:19.089412 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:19.089396    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-79lct-772qs"
Nov 02 09:38:19.089412 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:19.089419    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d8ba2fad85955f6f79f960f3c0ebd40962f89a662d7ecf13fb47773c3c22803"
Nov 02 09:38:39.634549 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:38:39.634503    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:38:44.936165 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:44.936108    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"]
Nov 02 09:38:44.936842 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:44.936540    4614 cpu_manager.go:414] "containerMap: removing container" podUID="b0bfa1c8-8148-4bdb-abd6-cc24256fd371" containerName="insights-gathering"
Nov 02 09:38:44.936842 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:44.936559    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bfa1c8-8148-4bdb-abd6-cc24256fd371" containerName="insights-gathering"
Nov 02 09:38:44.936842 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:44.936660    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bfa1c8-8148-4bdb-abd6-cc24256fd371" containerName="insights-gathering"
Nov 02 09:38:44.939789 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:44.939773    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"
Nov 02 09:38:44.944185 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:44.944165    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 09:38:44.945294 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:44.945276    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 09:38:44.945401 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:44.945382    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2hq4x\""
Nov 02 09:38:44.949915 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:44.949895    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"]
Nov 02 09:38:44.986403 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:44.986377    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59cjr\" (UniqueName: \"kubernetes.io/projected/2f975a2f-576c-4cd0-95f3-40e866239edb-kube-api-access-59cjr\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q\" (UID: \"2f975a2f-576c-4cd0-95f3-40e866239edb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"
Nov 02 09:38:44.986538 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:44.986429    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f975a2f-576c-4cd0-95f3-40e866239edb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q\" (UID: \"2f975a2f-576c-4cd0-95f3-40e866239edb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"
Nov 02 09:38:44.986538 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:44.986490    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f975a2f-576c-4cd0-95f3-40e866239edb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q\" (UID: \"2f975a2f-576c-4cd0-95f3-40e866239edb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"
Nov 02 09:38:45.049991 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.049966    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"]
Nov 02 09:38:45.053295 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.053280    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"
Nov 02 09:38:45.062632 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.062608    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-x2wzr\""
Nov 02 09:38:45.063075 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.063050    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"]
Nov 02 09:38:45.086733 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.086704    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd4e229-689d-4708-b1fb-67b05d77fd37-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd\" (UID: \"2bd4e229-689d-4708-b1fb-67b05d77fd37\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"
Nov 02 09:38:45.086822 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.086739    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f975a2f-576c-4cd0-95f3-40e866239edb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q\" (UID: \"2f975a2f-576c-4cd0-95f3-40e866239edb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"
Nov 02 09:38:45.086822 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.086762    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f975a2f-576c-4cd0-95f3-40e866239edb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q\" (UID: \"2f975a2f-576c-4cd0-95f3-40e866239edb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"
Nov 02 09:38:45.086903 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.086822    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zpmj\" (UniqueName: \"kubernetes.io/projected/2bd4e229-689d-4708-b1fb-67b05d77fd37-kube-api-access-5zpmj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd\" (UID: \"2bd4e229-689d-4708-b1fb-67b05d77fd37\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"
Nov 02 09:38:45.086903 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.086863    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2bd4e229-689d-4708-b1fb-67b05d77fd37-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd\" (UID: \"2bd4e229-689d-4708-b1fb-67b05d77fd37\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"
Nov 02 09:38:45.086966 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.086907    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59cjr\" (UniqueName: \"kubernetes.io/projected/2f975a2f-576c-4cd0-95f3-40e866239edb-kube-api-access-59cjr\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q\" (UID: \"2f975a2f-576c-4cd0-95f3-40e866239edb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"
Nov 02 09:38:45.087109 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.087089    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f975a2f-576c-4cd0-95f3-40e866239edb-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q\" (UID: \"2f975a2f-576c-4cd0-95f3-40e866239edb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"
Nov 02 09:38:45.087177 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.087157    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f975a2f-576c-4cd0-95f3-40e866239edb-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q\" (UID: \"2f975a2f-576c-4cd0-95f3-40e866239edb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"
Nov 02 09:38:45.099117 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.099093    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59cjr\" (UniqueName: \"kubernetes.io/projected/2f975a2f-576c-4cd0-95f3-40e866239edb-kube-api-access-59cjr\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q\" (UID: \"2f975a2f-576c-4cd0-95f3-40e866239edb\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"
Nov 02 09:38:45.188126 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.188046    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd4e229-689d-4708-b1fb-67b05d77fd37-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd\" (UID: \"2bd4e229-689d-4708-b1fb-67b05d77fd37\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"
Nov 02 09:38:45.188126 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.188097    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zpmj\" (UniqueName: \"kubernetes.io/projected/2bd4e229-689d-4708-b1fb-67b05d77fd37-kube-api-access-5zpmj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd\" (UID: \"2bd4e229-689d-4708-b1fb-67b05d77fd37\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"
Nov 02 09:38:45.188335 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.188148    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2bd4e229-689d-4708-b1fb-67b05d77fd37-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd\" (UID: \"2bd4e229-689d-4708-b1fb-67b05d77fd37\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"
Nov 02 09:38:45.188520 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.188500    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd4e229-689d-4708-b1fb-67b05d77fd37-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd\" (UID: \"2bd4e229-689d-4708-b1fb-67b05d77fd37\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"
Nov 02 09:38:45.188584 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.188541    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2bd4e229-689d-4708-b1fb-67b05d77fd37-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd\" (UID: \"2bd4e229-689d-4708-b1fb-67b05d77fd37\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"
Nov 02 09:38:45.203426 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.203402    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zpmj\" (UniqueName: \"kubernetes.io/projected/2bd4e229-689d-4708-b1fb-67b05d77fd37-kube-api-access-5zpmj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd\" (UID: \"2bd4e229-689d-4708-b1fb-67b05d77fd37\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"
Nov 02 09:38:45.260980 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.260952    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"
Nov 02 09:38:45.369437 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.369403    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"
Nov 02 09:38:45.409389 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.409340    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"]
Nov 02 09:38:45.412827 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.412807    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.420117 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.420093    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 09:38:45.422964 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.422246    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8zrbm\""
Nov 02 09:38:45.422964 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.422304    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 09:38:45.422964 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.422346    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 09:38:45.422964 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.422404    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 09:38:45.430645 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.430563    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"]
Nov 02 09:38:45.490944 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.490902    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892ac505-5776-4642-b6f9-bb776186e86a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.490944 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.490948    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.491165 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.490968    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/892ac505-5776-4642-b6f9-bb776186e86a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.491165 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.491070    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.491165 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.491106    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/892ac505-5776-4642-b6f9-bb776186e86a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.491165 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.491123    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b88qj\" (UniqueName: \"kubernetes.io/projected/892ac505-5776-4642-b6f9-bb776186e86a-kube-api-access-b88qj\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.491165 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.491143    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.540300 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.540138    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"]
Nov 02 09:38:45.542388 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:38:45.542363    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f975a2f_576c_4cd0_95f3_40e866239edb.slice/crio-2ca20426bd7c4e932f5b97ebe13ac1f1afbf5d5ec6ce625378dbb9ced4f4cf68 WatchSource:0}: Error finding container 2ca20426bd7c4e932f5b97ebe13ac1f1afbf5d5ec6ce625378dbb9ced4f4cf68: Status 404 returned error can't find the container with id 2ca20426bd7c4e932f5b97ebe13ac1f1afbf5d5ec6ce625378dbb9ced4f4cf68
Nov 02 09:38:45.592225 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.592198    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892ac505-5776-4642-b6f9-bb776186e86a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.592332 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.592230    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.592332 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.592252    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/892ac505-5776-4642-b6f9-bb776186e86a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.592332 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.592272    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.592332 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.592306    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/892ac505-5776-4642-b6f9-bb776186e86a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.592332 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.592322    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b88qj\" (UniqueName: \"kubernetes.io/projected/892ac505-5776-4642-b6f9-bb776186e86a-kube-api-access-b88qj\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.592567 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.592343    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.593050 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.593003    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/892ac505-5776-4642-b6f9-bb776186e86a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.594616 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.594590    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.594695 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.594631    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/892ac505-5776-4642-b6f9-bb776186e86a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.594695 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.594632    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.594695 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.594667    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.604330 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.604309    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892ac505-5776-4642-b6f9-bb776186e86a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.604656 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.604638    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b88qj\" (UniqueName: \"kubernetes.io/projected/892ac505-5776-4642-b6f9-bb776186e86a-kube-api-access-b88qj\") pod \"istiod-openshift-gateway-7876bfcb74-4dlgl\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:45.644542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.644505    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"]
Nov 02 09:38:45.646992 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:38:45.646963    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd4e229_689d_4708_b1fb_67b05d77fd37.slice/crio-30bfdb66375ef8836de7ca4ec79a6ec48559aeaba01edf8271cdc5481d1cb59f WatchSource:0}: Error finding container 30bfdb66375ef8836de7ca4ec79a6ec48559aeaba01edf8271cdc5481d1cb59f: Status 404 returned error can't find the container with id 30bfdb66375ef8836de7ca4ec79a6ec48559aeaba01edf8271cdc5481d1cb59f
Nov 02 09:38:45.728026 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:45.727994    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:46.003812 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:46.003788    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"]
Nov 02 09:38:46.006230 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:38:46.006201    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892ac505_5776_4642_b6f9_bb776186e86a.slice/crio-be53dddc04a8c147e306dd24213d3ce34500557e8f2d3ca34b8819d889edabac WatchSource:0}: Error finding container be53dddc04a8c147e306dd24213d3ce34500557e8f2d3ca34b8819d889edabac: Status 404 returned error can't find the container with id be53dddc04a8c147e306dd24213d3ce34500557e8f2d3ca34b8819d889edabac
Nov 02 09:38:46.188476 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:46.188398    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd" event={"ID":"2bd4e229-689d-4708-b1fb-67b05d77fd37","Type":"ContainerStarted","Data":"30bfdb66375ef8836de7ca4ec79a6ec48559aeaba01edf8271cdc5481d1cb59f"}
Nov 02 09:38:46.189953 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:46.189907    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl" event={"ID":"892ac505-5776-4642-b6f9-bb776186e86a","Type":"ContainerStarted","Data":"be53dddc04a8c147e306dd24213d3ce34500557e8f2d3ca34b8819d889edabac"}
Nov 02 09:38:46.191340 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:46.191310    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q" event={"ID":"2f975a2f-576c-4cd0-95f3-40e866239edb","Type":"ContainerStarted","Data":"2ca20426bd7c4e932f5b97ebe13ac1f1afbf5d5ec6ce625378dbb9ced4f4cf68"}
Nov 02 09:38:49.663888 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:49.663852    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:38:49.664283 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:49.663929    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:38:50.212962 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:50.212915    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl" event={"ID":"892ac505-5776-4642-b6f9-bb776186e86a","Type":"ContainerStarted","Data":"662b1ef12f732e699f432030092103623cc35d10a2cc0e6e8534bfb8d663777f"}
Nov 02 09:38:50.213164 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:50.212981    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:50.215022 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:50.214985    4614 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-4dlgl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 09:38:50.215154 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:50.215049    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl" podUID="892ac505-5776-4642-b6f9-bb776186e86a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 09:38:50.239437 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:50.239345    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl" podStartSLOduration=1.584370774 podStartE2EDuration="5.239326845s" totalImagesPullingTime="3.654956071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:38:50.238834022 +0000 UTC m=+211.071492102" watchObservedRunningTime="2026-11-02 09:38:50.239326845 +0000 UTC m=+211.071984922"
Nov 02 09:38:51.218338 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:51.218304    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:38:53.229542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:53.229508    4614 generic.go:356] "Generic (PLEG): container finished" podID="2f975a2f-576c-4cd0-95f3-40e866239edb" containerID="9cccadd6de626881feec411470dc171eba4b1d84a59a4e180d5fb8fbcb9cdbd0" exitCode=0
Nov 02 09:38:53.230020 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:53.229593    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q" event={"ID":"2f975a2f-576c-4cd0-95f3-40e866239edb","Type":"ContainerDied","Data":"9cccadd6de626881feec411470dc171eba4b1d84a59a4e180d5fb8fbcb9cdbd0"}
Nov 02 09:38:53.231092 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:53.231069    4614 generic.go:356] "Generic (PLEG): container finished" podID="2bd4e229-689d-4708-b1fb-67b05d77fd37" containerID="5a38cb81b3ef8a0c04e47706f5a3227c0eb423198b0d6252df51e360f31d17c1" exitCode=0
Nov 02 09:38:53.231202 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:53.231094    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd" event={"ID":"2bd4e229-689d-4708-b1fb-67b05d77fd37","Type":"ContainerDied","Data":"5a38cb81b3ef8a0c04e47706f5a3227c0eb423198b0d6252df51e360f31d17c1"}
Nov 02 09:38:54.237976 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:54.237942    4614 generic.go:356] "Generic (PLEG): container finished" podID="2bd4e229-689d-4708-b1fb-67b05d77fd37" containerID="66367d88f2c268311152f5aff5e6cd708c6be89ee009500213ef4240ce0eb3bb" exitCode=0
Nov 02 09:38:54.238486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:54.238034    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd" event={"ID":"2bd4e229-689d-4708-b1fb-67b05d77fd37","Type":"ContainerDied","Data":"66367d88f2c268311152f5aff5e6cd708c6be89ee009500213ef4240ce0eb3bb"}
Nov 02 09:38:56.249617 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:56.249560    4614 generic.go:356] "Generic (PLEG): container finished" podID="2f975a2f-576c-4cd0-95f3-40e866239edb" containerID="5364ab1429ba5526d22d0b92eb83e387c124f809cd44bfbcd31ae97a66652d63" exitCode=0
Nov 02 09:38:56.250138 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:38:56.249630    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q" event={"ID":"2f975a2f-576c-4cd0-95f3-40e866239edb","Type":"ContainerDied","Data":"5364ab1429ba5526d22d0b92eb83e387c124f809cd44bfbcd31ae97a66652d63"}
Nov 02 09:39:01.275699 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:01.275665    4614 generic.go:356] "Generic (PLEG): container finished" podID="2bd4e229-689d-4708-b1fb-67b05d77fd37" containerID="0937d8fea345d93bdf84ea6a447e127ab91fb344f70dbdadbf11f2ef993946e4" exitCode=0
Nov 02 09:39:01.276075 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:01.275753    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd" event={"ID":"2bd4e229-689d-4708-b1fb-67b05d77fd37","Type":"ContainerDied","Data":"0937d8fea345d93bdf84ea6a447e127ab91fb344f70dbdadbf11f2ef993946e4"}
Nov 02 09:39:02.419085 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:02.419052    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"
Nov 02 09:39:02.560151 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:02.560067    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd4e229-689d-4708-b1fb-67b05d77fd37-kube-api-access-5zpmj\" (UniqueName: \"kubernetes.io/projected/2bd4e229-689d-4708-b1fb-67b05d77fd37-kube-api-access-5zpmj\") pod \"2bd4e229-689d-4708-b1fb-67b05d77fd37\" (UID: \"2bd4e229-689d-4708-b1fb-67b05d77fd37\") "
Nov 02 09:39:02.560301 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:02.560155    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd4e229-689d-4708-b1fb-67b05d77fd37-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd4e229-689d-4708-b1fb-67b05d77fd37-bundle\") pod \"2bd4e229-689d-4708-b1fb-67b05d77fd37\" (UID: \"2bd4e229-689d-4708-b1fb-67b05d77fd37\") "
Nov 02 09:39:02.560301 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:02.560201    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd4e229-689d-4708-b1fb-67b05d77fd37-util\" (UniqueName: \"kubernetes.io/empty-dir/2bd4e229-689d-4708-b1fb-67b05d77fd37-util\") pod \"2bd4e229-689d-4708-b1fb-67b05d77fd37\" (UID: \"2bd4e229-689d-4708-b1fb-67b05d77fd37\") "
Nov 02 09:39:02.560929 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:02.560896    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd4e229-689d-4708-b1fb-67b05d77fd37-bundle" pod "2bd4e229-689d-4708-b1fb-67b05d77fd37" (UID: "2bd4e229-689d-4708-b1fb-67b05d77fd37"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:02.562662 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:02.562629    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd4e229-689d-4708-b1fb-67b05d77fd37-kube-api-access-5zpmj" pod "2bd4e229-689d-4708-b1fb-67b05d77fd37" (UID: "2bd4e229-689d-4708-b1fb-67b05d77fd37"). InnerVolumeSpecName "kube-api-access-5zpmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:39:02.566989 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:02.566964    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd4e229-689d-4708-b1fb-67b05d77fd37-util" pod "2bd4e229-689d-4708-b1fb-67b05d77fd37" (UID: "2bd4e229-689d-4708-b1fb-67b05d77fd37"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:02.661168 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:02.661128    4614 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd4e229-689d-4708-b1fb-67b05d77fd37-bundle\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:02.661168 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:02.661158    4614 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2bd4e229-689d-4708-b1fb-67b05d77fd37-util\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:02.661168 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:02.661168    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zpmj\" (UniqueName: \"kubernetes.io/projected/2bd4e229-689d-4708-b1fb-67b05d77fd37-kube-api-access-5zpmj\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:03.286923 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:03.286885    4614 generic.go:356] "Generic (PLEG): container finished" podID="2f975a2f-576c-4cd0-95f3-40e866239edb" containerID="76785a83a0bedb236eb4e028ce9573e63a86506031d610bf39bfaddcca037cdb" exitCode=0
Nov 02 09:39:03.287087 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:03.286973    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q" event={"ID":"2f975a2f-576c-4cd0-95f3-40e866239edb","Type":"ContainerDied","Data":"76785a83a0bedb236eb4e028ce9573e63a86506031d610bf39bfaddcca037cdb"}
Nov 02 09:39:03.288743 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:03.288716    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd" event={"ID":"2bd4e229-689d-4708-b1fb-67b05d77fd37","Type":"ContainerDied","Data":"30bfdb66375ef8836de7ca4ec79a6ec48559aeaba01edf8271cdc5481d1cb59f"}
Nov 02 09:39:03.288840 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:03.288754    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30bfdb66375ef8836de7ca4ec79a6ec48559aeaba01edf8271cdc5481d1cb59f"
Nov 02 09:39:03.288840 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:03.288729    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714mhgd"
Nov 02 09:39:04.411286 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:04.411264    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"
Nov 02 09:39:04.478041 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:04.478013    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f975a2f-576c-4cd0-95f3-40e866239edb-util\" (UniqueName: \"kubernetes.io/empty-dir/2f975a2f-576c-4cd0-95f3-40e866239edb-util\") pod \"2f975a2f-576c-4cd0-95f3-40e866239edb\" (UID: \"2f975a2f-576c-4cd0-95f3-40e866239edb\") "
Nov 02 09:39:04.478183 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:04.478086    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f975a2f-576c-4cd0-95f3-40e866239edb-kube-api-access-59cjr\" (UniqueName: \"kubernetes.io/projected/2f975a2f-576c-4cd0-95f3-40e866239edb-kube-api-access-59cjr\") pod \"2f975a2f-576c-4cd0-95f3-40e866239edb\" (UID: \"2f975a2f-576c-4cd0-95f3-40e866239edb\") "
Nov 02 09:39:04.478183 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:04.478113    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f975a2f-576c-4cd0-95f3-40e866239edb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f975a2f-576c-4cd0-95f3-40e866239edb-bundle\") pod \"2f975a2f-576c-4cd0-95f3-40e866239edb\" (UID: \"2f975a2f-576c-4cd0-95f3-40e866239edb\") "
Nov 02 09:39:04.478763 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:04.478731    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f975a2f-576c-4cd0-95f3-40e866239edb-bundle" pod "2f975a2f-576c-4cd0-95f3-40e866239edb" (UID: "2f975a2f-576c-4cd0-95f3-40e866239edb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:04.480135 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:04.480110    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f975a2f-576c-4cd0-95f3-40e866239edb-kube-api-access-59cjr" pod "2f975a2f-576c-4cd0-95f3-40e866239edb" (UID: "2f975a2f-576c-4cd0-95f3-40e866239edb"). InnerVolumeSpecName "kube-api-access-59cjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:39:04.481985 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:04.481961    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f975a2f-576c-4cd0-95f3-40e866239edb-util" pod "2f975a2f-576c-4cd0-95f3-40e866239edb" (UID: "2f975a2f-576c-4cd0-95f3-40e866239edb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:39:04.579609 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:04.579537    4614 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f975a2f-576c-4cd0-95f3-40e866239edb-bundle\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:04.579609 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:04.579562    4614 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f975a2f-576c-4cd0-95f3-40e866239edb-util\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:04.579609 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:04.579571    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59cjr\" (UniqueName: \"kubernetes.io/projected/2f975a2f-576c-4cd0-95f3-40e866239edb-kube-api-access-59cjr\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:39:05.299871 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:05.299830    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q" event={"ID":"2f975a2f-576c-4cd0-95f3-40e866239edb","Type":"ContainerDied","Data":"2ca20426bd7c4e932f5b97ebe13ac1f1afbf5d5ec6ce625378dbb9ced4f4cf68"}
Nov 02 09:39:05.299871 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:05.299851    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csbv9q"
Nov 02 09:39:05.300081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:05.299900    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ca20426bd7c4e932f5b97ebe13ac1f1afbf5d5ec6ce625378dbb9ced4f4cf68"
Nov 02 09:39:09.634524 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:39:09.634479    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:39:24.450744 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.450710    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"]
Nov 02 09:39:24.451414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.451126    4614 cpu_manager.go:414] "containerMap: removing container" podUID="2bd4e229-689d-4708-b1fb-67b05d77fd37" containerName="extract"
Nov 02 09:39:24.451414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.451145    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd4e229-689d-4708-b1fb-67b05d77fd37" containerName="extract"
Nov 02 09:39:24.451414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.451182    4614 cpu_manager.go:414] "containerMap: removing container" podUID="2f975a2f-576c-4cd0-95f3-40e866239edb" containerName="util"
Nov 02 09:39:24.451414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.451190    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f975a2f-576c-4cd0-95f3-40e866239edb" containerName="util"
Nov 02 09:39:24.451414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.451202    4614 cpu_manager.go:414] "containerMap: removing container" podUID="2bd4e229-689d-4708-b1fb-67b05d77fd37" containerName="pull"
Nov 02 09:39:24.451414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.451210    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd4e229-689d-4708-b1fb-67b05d77fd37" containerName="pull"
Nov 02 09:39:24.451414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.451223    4614 cpu_manager.go:414] "containerMap: removing container" podUID="2f975a2f-576c-4cd0-95f3-40e866239edb" containerName="extract"
Nov 02 09:39:24.451414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.451230    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f975a2f-576c-4cd0-95f3-40e866239edb" containerName="extract"
Nov 02 09:39:24.451414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.451242    4614 cpu_manager.go:414] "containerMap: removing container" podUID="2bd4e229-689d-4708-b1fb-67b05d77fd37" containerName="util"
Nov 02 09:39:24.451414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.451249    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd4e229-689d-4708-b1fb-67b05d77fd37" containerName="util"
Nov 02 09:39:24.451414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.451260    4614 cpu_manager.go:414] "containerMap: removing container" podUID="2f975a2f-576c-4cd0-95f3-40e866239edb" containerName="pull"
Nov 02 09:39:24.451414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.451268    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f975a2f-576c-4cd0-95f3-40e866239edb" containerName="pull"
Nov 02 09:39:24.451414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.451373    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f975a2f-576c-4cd0-95f3-40e866239edb" containerName="extract"
Nov 02 09:39:24.451414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.451386    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd4e229-689d-4708-b1fb-67b05d77fd37" containerName="extract"
Nov 02 09:39:24.456647 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.456625    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.481744 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.481715    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"]
Nov 02 09:39:24.535341 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.535308    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-ptvwz\""
Nov 02 09:39:24.535559 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.535426    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 09:39:24.535559 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.535427    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 09:39:24.535559 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.535447    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 09:39:24.535559 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.535462    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 09:39:24.535559 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.535518    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 09:39:24.632714 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.632681    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.632714 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.632715    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-tmp\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.632909 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.632731    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bftjk\" (UniqueName: \"kubernetes.io/projected/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-kube-api-access-bftjk\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.632909 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.632868    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.632909 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.632892    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-manifests\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.633019 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.632914    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.733689 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.733661    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.733818 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.733697    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-manifests\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.733818 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.733720    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.733818 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.733753    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.733818 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.733769    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-tmp\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.733818 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.733785    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bftjk\" (UniqueName: \"kubernetes.io/projected/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-kube-api-access-bftjk\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.734075 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.734052    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-manifests\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.736107 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.736082    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-tmp\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.736379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.736336    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.736458 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.736411    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.736458 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.736412    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.746093 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.746072    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bftjk\" (UniqueName: \"kubernetes.io/projected/25ea051f-bf6f-4b0e-ac1e-91ff591f659f-kube-api-access-bftjk\") pod \"opendatahub-operator-controller-manager-f7698d755-k98vj\" (UID: \"25ea051f-bf6f-4b0e-ac1e-91ff591f659f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:24.768698 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:24.768678    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:25.067199 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:25.066999    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"]
Nov 02 09:39:25.070145 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:39:25.070106    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ea051f_bf6f_4b0e_ac1e_91ff591f659f.slice/crio-a155f3b10f0fe5a23a9562be48a6f2e90fae166449261484dcebde46ce59f20a WatchSource:0}: Error finding container a155f3b10f0fe5a23a9562be48a6f2e90fae166449261484dcebde46ce59f20a: Status 404 returned error can't find the container with id a155f3b10f0fe5a23a9562be48a6f2e90fae166449261484dcebde46ce59f20a
Nov 02 09:39:25.381520 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:25.381439    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj" event={"ID":"25ea051f-bf6f-4b0e-ac1e-91ff591f659f","Type":"ContainerStarted","Data":"a155f3b10f0fe5a23a9562be48a6f2e90fae166449261484dcebde46ce59f20a"}
Nov 02 09:39:28.397189 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:28.397147    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj" event={"ID":"25ea051f-bf6f-4b0e-ac1e-91ff591f659f","Type":"ContainerStarted","Data":"1ad7778635ea2c0b1dd9629bd056dcca420a4abb2aad1dab3d7903de391d57b0"}
Nov 02 09:39:29.403254 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:29.403217    4614 generic.go:356] "Generic (PLEG): container finished" podID="25ea051f-bf6f-4b0e-ac1e-91ff591f659f" containerID="1ad7778635ea2c0b1dd9629bd056dcca420a4abb2aad1dab3d7903de391d57b0" exitCode=0
Nov 02 09:39:29.403766 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:29.403308    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj" event={"ID":"25ea051f-bf6f-4b0e-ac1e-91ff591f659f","Type":"ContainerDied","Data":"1ad7778635ea2c0b1dd9629bd056dcca420a4abb2aad1dab3d7903de391d57b0"}
Nov 02 09:39:30.409825 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:30.409792    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj" event={"ID":"25ea051f-bf6f-4b0e-ac1e-91ff591f659f","Type":"ContainerStarted","Data":"ac69338127cdd1af99f72a1a274f8a5cdec2dc73d93d577afc4b9f6d05e7db43"}
Nov 02 09:39:30.410234 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:30.409889    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:30.453484 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:30.453432    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj" podStartSLOduration=2.471754738 podStartE2EDuration="6.453417231s" totalImagesPullingTime="3.49406165s" totalInitContainerRuntime="487.600843ms" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:39:30.45141404 +0000 UTC m=+251.284072121" watchObservedRunningTime="2026-11-02 09:39:30.453417231 +0000 UTC m=+251.286075311"
Nov 02 09:39:39.634965 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:39:39.634925    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:39:41.416712 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:41.416686    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7698d755-k98vj"
Nov 02 09:39:56.562834 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.562798    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c759d848c-77mmj"]
Nov 02 09:39:56.567582 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.567558    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c759d848c-77mmj"
Nov 02 09:39:56.578559 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.578537    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 09:39:56.578728 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.578597    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5cbfk\""
Nov 02 09:39:56.580487 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.578725    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 09:39:56.581548 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.581522    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c759d848c-77mmj"]
Nov 02 09:39:56.687044 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.687003    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b215a15b-8411-4d37-af02-20819dcb01c9-tls-certs\") pod \"kube-auth-proxy-7c759d848c-77mmj\" (UID: \"b215a15b-8411-4d37-af02-20819dcb01c9\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-77mmj"
Nov 02 09:39:56.687044 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.687041    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhnsz\" (UniqueName: \"kubernetes.io/projected/b215a15b-8411-4d37-af02-20819dcb01c9-kube-api-access-fhnsz\") pod \"kube-auth-proxy-7c759d848c-77mmj\" (UID: \"b215a15b-8411-4d37-af02-20819dcb01c9\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-77mmj"
Nov 02 09:39:56.687238 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.687079    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b215a15b-8411-4d37-af02-20819dcb01c9-tmp\") pod \"kube-auth-proxy-7c759d848c-77mmj\" (UID: \"b215a15b-8411-4d37-af02-20819dcb01c9\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-77mmj"
Nov 02 09:39:56.787733 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.787705    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b215a15b-8411-4d37-af02-20819dcb01c9-tls-certs\") pod \"kube-auth-proxy-7c759d848c-77mmj\" (UID: \"b215a15b-8411-4d37-af02-20819dcb01c9\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-77mmj"
Nov 02 09:39:56.787733 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.787738    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhnsz\" (UniqueName: \"kubernetes.io/projected/b215a15b-8411-4d37-af02-20819dcb01c9-kube-api-access-fhnsz\") pod \"kube-auth-proxy-7c759d848c-77mmj\" (UID: \"b215a15b-8411-4d37-af02-20819dcb01c9\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-77mmj"
Nov 02 09:39:56.787944 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.787770    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b215a15b-8411-4d37-af02-20819dcb01c9-tmp\") pod \"kube-auth-proxy-7c759d848c-77mmj\" (UID: \"b215a15b-8411-4d37-af02-20819dcb01c9\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-77mmj"
Nov 02 09:39:56.790031 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.790008    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b215a15b-8411-4d37-af02-20819dcb01c9-tmp\") pod \"kube-auth-proxy-7c759d848c-77mmj\" (UID: \"b215a15b-8411-4d37-af02-20819dcb01c9\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-77mmj"
Nov 02 09:39:56.790286 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.790269    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b215a15b-8411-4d37-af02-20819dcb01c9-tls-certs\") pod \"kube-auth-proxy-7c759d848c-77mmj\" (UID: \"b215a15b-8411-4d37-af02-20819dcb01c9\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-77mmj"
Nov 02 09:39:56.800185 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.800162    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhnsz\" (UniqueName: \"kubernetes.io/projected/b215a15b-8411-4d37-af02-20819dcb01c9-kube-api-access-fhnsz\") pod \"kube-auth-proxy-7c759d848c-77mmj\" (UID: \"b215a15b-8411-4d37-af02-20819dcb01c9\") " pod="openshift-ingress/kube-auth-proxy-7c759d848c-77mmj"
Nov 02 09:39:56.883415 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:56.883318    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c759d848c-77mmj"
Nov 02 09:39:57.169935 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:57.169806    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c759d848c-77mmj"]
Nov 02 09:39:57.173318 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:39:57.173291    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb215a15b_8411_4d37_af02_20819dcb01c9.slice/crio-d7d1abef74e6d8b49618f9bcb7c5727747b3068947a16bd07dd7f6abeba00418 WatchSource:0}: Error finding container d7d1abef74e6d8b49618f9bcb7c5727747b3068947a16bd07dd7f6abeba00418: Status 404 returned error can't find the container with id d7d1abef74e6d8b49618f9bcb7c5727747b3068947a16bd07dd7f6abeba00418
Nov 02 09:39:57.526996 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:39:57.526959    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c759d848c-77mmj" event={"ID":"b215a15b-8411-4d37-af02-20819dcb01c9","Type":"ContainerStarted","Data":"d7d1abef74e6d8b49618f9bcb7c5727747b3068947a16bd07dd7f6abeba00418"}
Nov 02 09:40:00.544463 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:00.544416    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c759d848c-77mmj" event={"ID":"b215a15b-8411-4d37-af02-20819dcb01c9","Type":"ContainerStarted","Data":"f6f0771444dc60ef7a3c5ae531946704b201c2e3bad253f028f335e71164b9f3"}
Nov 02 09:40:00.562072 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:00.562024    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c759d848c-77mmj" podStartSLOduration=1.6269890390000001 podStartE2EDuration="4.562012234s" totalImagesPullingTime="2.935023195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:00.561473501 +0000 UTC m=+281.394131582" watchObservedRunningTime="2026-11-02 09:40:00.562012234 +0000 UTC m=+281.394670360"
Nov 02 09:40:09.635132 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:40:09.635099    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:40:17.758008 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:17.757974    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-svc5b"]
Nov 02 09:40:17.761520 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:17.761500    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b"
Nov 02 09:40:17.770057 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:17.770037    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zj9w7\""
Nov 02 09:40:17.771438 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:17.771404    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-svc5b"]
Nov 02 09:40:17.786262 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:17.786236    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np567\" (UniqueName: \"kubernetes.io/projected/80d4a3f0-8f12-4539-984c-2d97af2a15b9-kube-api-access-np567\") pod \"limitador-limitador-74dfcfb894-svc5b\" (UID: \"80d4a3f0-8f12-4539-984c-2d97af2a15b9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b"
Nov 02 09:40:17.786499 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:17.786474    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80d4a3f0-8f12-4539-984c-2d97af2a15b9-config-file\") pod \"limitador-limitador-74dfcfb894-svc5b\" (UID: \"80d4a3f0-8f12-4539-984c-2d97af2a15b9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b"
Nov 02 09:40:17.838086 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:17.838050    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 09:40:17.838251 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:17.838137    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 09:40:17.838251 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:17.838148    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 09:40:17.858113 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:17.858081    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-svc5b"]
Nov 02 09:40:17.887403 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:17.887375    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np567\" (UniqueName: \"kubernetes.io/projected/80d4a3f0-8f12-4539-984c-2d97af2a15b9-kube-api-access-np567\") pod \"limitador-limitador-74dfcfb894-svc5b\" (UID: \"80d4a3f0-8f12-4539-984c-2d97af2a15b9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b"
Nov 02 09:40:17.887505 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:17.887413    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80d4a3f0-8f12-4539-984c-2d97af2a15b9-config-file\") pod \"limitador-limitador-74dfcfb894-svc5b\" (UID: \"80d4a3f0-8f12-4539-984c-2d97af2a15b9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b"
Nov 02 09:40:17.887976 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:17.887960    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80d4a3f0-8f12-4539-984c-2d97af2a15b9-config-file\") pod \"limitador-limitador-74dfcfb894-svc5b\" (UID: \"80d4a3f0-8f12-4539-984c-2d97af2a15b9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b"
Nov 02 09:40:17.915173 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:17.915145    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np567\" (UniqueName: \"kubernetes.io/projected/80d4a3f0-8f12-4539-984c-2d97af2a15b9-kube-api-access-np567\") pod \"limitador-limitador-74dfcfb894-svc5b\" (UID: \"80d4a3f0-8f12-4539-984c-2d97af2a15b9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b"
Nov 02 09:40:18.075536 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:18.075455    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b"
Nov 02 09:40:18.348641 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:18.348607    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-svc5b"]
Nov 02 09:40:18.350872 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:40:18.350845    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d4a3f0_8f12_4539_984c_2d97af2a15b9.slice/crio-897565df4e34dc45faa0f22c152660d42ec582ab2aa8e14a1582841122f885e6 WatchSource:0}: Error finding container 897565df4e34dc45faa0f22c152660d42ec582ab2aa8e14a1582841122f885e6: Status 404 returned error can't find the container with id 897565df4e34dc45faa0f22c152660d42ec582ab2aa8e14a1582841122f885e6
Nov 02 09:40:18.619308 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:18.619219    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b" event={"ID":"80d4a3f0-8f12-4539-984c-2d97af2a15b9","Type":"ContainerStarted","Data":"897565df4e34dc45faa0f22c152660d42ec582ab2aa8e14a1582841122f885e6"}
Nov 02 09:40:19.329750 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:19.329717    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 09:40:19.330275 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:19.330254    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 09:40:22.637688 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:22.637652    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b" event={"ID":"80d4a3f0-8f12-4539-984c-2d97af2a15b9","Type":"ContainerStarted","Data":"6edcd9c747b7be9e849d59d335154846cb791ce795e756809b1eaf803d65dd06"}
Nov 02 09:40:22.638070 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:22.637748    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b"
Nov 02 09:40:22.658057 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:22.658008    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b" podStartSLOduration=2.385007835 podStartE2EDuration="5.657995517s" totalImagesPullingTime="3.272987682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:22.656578659 +0000 UTC m=+303.489236740" watchObservedRunningTime="2026-11-02 09:40:22.657995517 +0000 UTC m=+303.490653597"
Nov 02 09:40:31.649015 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:31.648935    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-svc5b"]
Nov 02 09:40:31.649651 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:31.649138    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b" podUID="80d4a3f0-8f12-4539-984c-2d97af2a15b9" containerName="limitador" containerID="cri-o://6edcd9c747b7be9e849d59d335154846cb791ce795e756809b1eaf803d65dd06" gracePeriod=30
Nov 02 09:40:31.650473 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:31.650445    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b"
Nov 02 09:40:32.185950 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.185925    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b"
Nov 02 09:40:32.293548 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.293519    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d4a3f0-8f12-4539-984c-2d97af2a15b9-config-file\" (UniqueName: \"kubernetes.io/configmap/80d4a3f0-8f12-4539-984c-2d97af2a15b9-config-file\") pod \"80d4a3f0-8f12-4539-984c-2d97af2a15b9\" (UID: \"80d4a3f0-8f12-4539-984c-2d97af2a15b9\") "
Nov 02 09:40:32.293706 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.293575    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d4a3f0-8f12-4539-984c-2d97af2a15b9-kube-api-access-np567\" (UniqueName: \"kubernetes.io/projected/80d4a3f0-8f12-4539-984c-2d97af2a15b9-kube-api-access-np567\") pod \"80d4a3f0-8f12-4539-984c-2d97af2a15b9\" (UID: \"80d4a3f0-8f12-4539-984c-2d97af2a15b9\") "
Nov 02 09:40:32.293865 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.293844    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d4a3f0-8f12-4539-984c-2d97af2a15b9-config-file" pod "80d4a3f0-8f12-4539-984c-2d97af2a15b9" (UID: "80d4a3f0-8f12-4539-984c-2d97af2a15b9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:40:32.295772 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.295749    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d4a3f0-8f12-4539-984c-2d97af2a15b9-kube-api-access-np567" pod "80d4a3f0-8f12-4539-984c-2d97af2a15b9" (UID: "80d4a3f0-8f12-4539-984c-2d97af2a15b9"). InnerVolumeSpecName "kube-api-access-np567". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:40:32.394401 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.394336    4614 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80d4a3f0-8f12-4539-984c-2d97af2a15b9-config-file\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:40:32.394401 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.394392    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np567\" (UniqueName: \"kubernetes.io/projected/80d4a3f0-8f12-4539-984c-2d97af2a15b9-kube-api-access-np567\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:40:32.678824 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.678731    4614 generic.go:356] "Generic (PLEG): container finished" podID="80d4a3f0-8f12-4539-984c-2d97af2a15b9" containerID="6edcd9c747b7be9e849d59d335154846cb791ce795e756809b1eaf803d65dd06" exitCode=0
Nov 02 09:40:32.678824 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.678781    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b"
Nov 02 09:40:32.678824 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.678810    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b" event={"ID":"80d4a3f0-8f12-4539-984c-2d97af2a15b9","Type":"ContainerDied","Data":"6edcd9c747b7be9e849d59d335154846cb791ce795e756809b1eaf803d65dd06"}
Nov 02 09:40:32.679301 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.678865    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-svc5b" event={"ID":"80d4a3f0-8f12-4539-984c-2d97af2a15b9","Type":"ContainerDied","Data":"897565df4e34dc45faa0f22c152660d42ec582ab2aa8e14a1582841122f885e6"}
Nov 02 09:40:32.679301 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.678886    4614 scope.go:122] "RemoveContainer" containerID="6edcd9c747b7be9e849d59d335154846cb791ce795e756809b1eaf803d65dd06"
Nov 02 09:40:32.687393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.687376    4614 scope.go:122] "RemoveContainer" containerID="6edcd9c747b7be9e849d59d335154846cb791ce795e756809b1eaf803d65dd06"
Nov 02 09:40:32.687650 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:40:32.687631    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6edcd9c747b7be9e849d59d335154846cb791ce795e756809b1eaf803d65dd06\": container with ID starting with 6edcd9c747b7be9e849d59d335154846cb791ce795e756809b1eaf803d65dd06 not found: ID does not exist" containerID="6edcd9c747b7be9e849d59d335154846cb791ce795e756809b1eaf803d65dd06"
Nov 02 09:40:32.687695 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.687658    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6edcd9c747b7be9e849d59d335154846cb791ce795e756809b1eaf803d65dd06"} err="failed to get container status \"6edcd9c747b7be9e849d59d335154846cb791ce795e756809b1eaf803d65dd06\": rpc error: code = NotFound desc = could not find container \"6edcd9c747b7be9e849d59d335154846cb791ce795e756809b1eaf803d65dd06\": container with ID starting with 6edcd9c747b7be9e849d59d335154846cb791ce795e756809b1eaf803d65dd06 not found: ID does not exist"
Nov 02 09:40:32.700453 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.700427    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-svc5b"]
Nov 02 09:40:32.706367 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:32.706337    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-svc5b"]
Nov 02 09:40:33.318217 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:33.318181    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d4a3f0-8f12-4539-984c-2d97af2a15b9" path="/var/lib/kubelet/pods/80d4a3f0-8f12-4539-984c-2d97af2a15b9/volumes"
Nov 02 09:40:39.635819 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:40:39.635779    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:40:45.448622 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.448584    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc"]
Nov 02 09:40:45.449100 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.449026    4614 cpu_manager.go:414] "containerMap: removing container" podUID="80d4a3f0-8f12-4539-984c-2d97af2a15b9" containerName="limitador"
Nov 02 09:40:45.449100 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.449042    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d4a3f0-8f12-4539-984c-2d97af2a15b9" containerName="limitador"
Nov 02 09:40:45.449212 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.449159    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d4a3f0-8f12-4539-984c-2d97af2a15b9" containerName="limitador"
Nov 02 09:40:45.454579 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.454559    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc"
Nov 02 09:40:45.459660 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.459637    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 09:40:45.460562 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.460508    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 02 09:40:45.460755 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.460717    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-b54xl\""
Nov 02 09:40:45.460842 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.460802    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 09:40:45.461513 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.461415    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc"]
Nov 02 09:40:45.492904 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.492871    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c0cc5dd2-053c-4a5b-8c6b-001ea092716e-data\") pod \"postgres-7cdbf6dc44-sd2jc\" (UID: \"c0cc5dd2-053c-4a5b-8c6b-001ea092716e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc"
Nov 02 09:40:45.493103 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.492931    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5bng\" (UniqueName: \"kubernetes.io/projected/c0cc5dd2-053c-4a5b-8c6b-001ea092716e-kube-api-access-q5bng\") pod \"postgres-7cdbf6dc44-sd2jc\" (UID: \"c0cc5dd2-053c-4a5b-8c6b-001ea092716e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc"
Nov 02 09:40:45.593868 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.593833    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5bng\" (UniqueName: \"kubernetes.io/projected/c0cc5dd2-053c-4a5b-8c6b-001ea092716e-kube-api-access-q5bng\") pod \"postgres-7cdbf6dc44-sd2jc\" (UID: \"c0cc5dd2-053c-4a5b-8c6b-001ea092716e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc"
Nov 02 09:40:45.594044 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.593907    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c0cc5dd2-053c-4a5b-8c6b-001ea092716e-data\") pod \"postgres-7cdbf6dc44-sd2jc\" (UID: \"c0cc5dd2-053c-4a5b-8c6b-001ea092716e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc"
Nov 02 09:40:45.594281 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.594260    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c0cc5dd2-053c-4a5b-8c6b-001ea092716e-data\") pod \"postgres-7cdbf6dc44-sd2jc\" (UID: \"c0cc5dd2-053c-4a5b-8c6b-001ea092716e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc"
Nov 02 09:40:45.606510 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.606485    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5bng\" (UniqueName: \"kubernetes.io/projected/c0cc5dd2-053c-4a5b-8c6b-001ea092716e-kube-api-access-q5bng\") pod \"postgres-7cdbf6dc44-sd2jc\" (UID: \"c0cc5dd2-053c-4a5b-8c6b-001ea092716e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc"
Nov 02 09:40:45.768533 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:45.768498    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc"
Nov 02 09:40:46.041699 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:46.041592    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc"]
Nov 02 09:40:46.044458 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:40:46.044428    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cc5dd2_053c_4a5b_8c6b_001ea092716e.slice/crio-0323c7f204ab75b667a6ce6f156ce3ae04673d5f7b4f61872f83f20d3ca13a6f WatchSource:0}: Error finding container 0323c7f204ab75b667a6ce6f156ce3ae04673d5f7b4f61872f83f20d3ca13a6f: Status 404 returned error can't find the container with id 0323c7f204ab75b667a6ce6f156ce3ae04673d5f7b4f61872f83f20d3ca13a6f
Nov 02 09:40:46.045652 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:46.045634    4614 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 09:40:46.744512 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:46.744476    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc" event={"ID":"c0cc5dd2-053c-4a5b-8c6b-001ea092716e","Type":"ContainerStarted","Data":"0323c7f204ab75b667a6ce6f156ce3ae04673d5f7b4f61872f83f20d3ca13a6f"}
Nov 02 09:40:52.779004 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:52.778964    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc" event={"ID":"c0cc5dd2-053c-4a5b-8c6b-001ea092716e","Type":"ContainerStarted","Data":"fc345603a5fbbf4061c227d28eb7a8b474162fa419853229d2db0e9db40746ad"}
Nov 02 09:40:52.780173 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:52.780146    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc"
Nov 02 09:40:52.800895 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:52.800845    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc" podStartSLOduration=2.079637008 podStartE2EDuration="7.800830894s" totalImagesPullingTime="5.721193886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:52.800184841 +0000 UTC m=+333.632842912" watchObservedRunningTime="2026-11-02 09:40:52.800830894 +0000 UTC m=+333.633488974"
Nov 02 09:40:54.218678 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.218644    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-cfpwn"]
Nov 02 09:40:54.221963 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.221944    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-cfpwn"
Nov 02 09:40:54.225744 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.225725    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6vkxp\""
Nov 02 09:40:54.229084 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.229060    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 09:40:54.230954 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.230932    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"]
Nov 02 09:40:54.232149 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.232136    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 09:40:54.234053 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.234035    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"
Nov 02 09:40:54.234684 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.234589    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 02 09:40:54.235611 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.235592    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-cfpwn"]
Nov 02 09:40:54.245657 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.245628    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"]
Nov 02 09:40:54.283618 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.283586    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5kq\" (UniqueName: \"kubernetes.io/projected/b0b53ec3-a9e4-45d7-8aa4-59237abc2614-kube-api-access-9j5kq\") pod \"kserve-controller-manager-776cc66d47-cfpwn\" (UID: \"b0b53ec3-a9e4-45d7-8aa4-59237abc2614\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cfpwn"
Nov 02 09:40:54.283750 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.283629    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0b53ec3-a9e4-45d7-8aa4-59237abc2614-cert\") pod \"kserve-controller-manager-776cc66d47-cfpwn\" (UID: \"b0b53ec3-a9e4-45d7-8aa4-59237abc2614\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cfpwn"
Nov 02 09:40:54.300114 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.300075    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 02 09:40:54.300250 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.300195    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-8czpp\""
Nov 02 09:40:54.384480 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.384445    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0b53ec3-a9e4-45d7-8aa4-59237abc2614-cert\") pod \"kserve-controller-manager-776cc66d47-cfpwn\" (UID: \"b0b53ec3-a9e4-45d7-8aa4-59237abc2614\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cfpwn"
Nov 02 09:40:54.384622 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.384498    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a1a4483-017b-4cb2-84e4-910f30499440-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lkqpk\" (UID: \"6a1a4483-017b-4cb2-84e4-910f30499440\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"
Nov 02 09:40:54.384622 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.384546    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5kq\" (UniqueName: \"kubernetes.io/projected/b0b53ec3-a9e4-45d7-8aa4-59237abc2614-kube-api-access-9j5kq\") pod \"kserve-controller-manager-776cc66d47-cfpwn\" (UID: \"b0b53ec3-a9e4-45d7-8aa4-59237abc2614\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cfpwn"
Nov 02 09:40:54.384622 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.384564    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpgkg\" (UniqueName: \"kubernetes.io/projected/6a1a4483-017b-4cb2-84e4-910f30499440-kube-api-access-lpgkg\") pod \"llmisvc-controller-manager-869d5dcc5d-lkqpk\" (UID: \"6a1a4483-017b-4cb2-84e4-910f30499440\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"
Nov 02 09:40:54.386900 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.386880    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0b53ec3-a9e4-45d7-8aa4-59237abc2614-cert\") pod \"kserve-controller-manager-776cc66d47-cfpwn\" (UID: \"b0b53ec3-a9e4-45d7-8aa4-59237abc2614\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cfpwn"
Nov 02 09:40:54.397034 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.397010    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5kq\" (UniqueName: \"kubernetes.io/projected/b0b53ec3-a9e4-45d7-8aa4-59237abc2614-kube-api-access-9j5kq\") pod \"kserve-controller-manager-776cc66d47-cfpwn\" (UID: \"b0b53ec3-a9e4-45d7-8aa4-59237abc2614\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cfpwn"
Nov 02 09:40:54.484932 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.484852    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpgkg\" (UniqueName: \"kubernetes.io/projected/6a1a4483-017b-4cb2-84e4-910f30499440-kube-api-access-lpgkg\") pod \"llmisvc-controller-manager-869d5dcc5d-lkqpk\" (UID: \"6a1a4483-017b-4cb2-84e4-910f30499440\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"
Nov 02 09:40:54.484932 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.484907    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a1a4483-017b-4cb2-84e4-910f30499440-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lkqpk\" (UID: \"6a1a4483-017b-4cb2-84e4-910f30499440\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"
Nov 02 09:40:54.487226 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.487202    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a1a4483-017b-4cb2-84e4-910f30499440-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lkqpk\" (UID: \"6a1a4483-017b-4cb2-84e4-910f30499440\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"
Nov 02 09:40:54.497060 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.497034    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpgkg\" (UniqueName: \"kubernetes.io/projected/6a1a4483-017b-4cb2-84e4-910f30499440-kube-api-access-lpgkg\") pod \"llmisvc-controller-manager-869d5dcc5d-lkqpk\" (UID: \"6a1a4483-017b-4cb2-84e4-910f30499440\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"
Nov 02 09:40:54.535475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.535441    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-cfpwn"
Nov 02 09:40:54.548609 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.548582    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"
Nov 02 09:40:54.567157 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.567127    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"]
Nov 02 09:40:54.572432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.572036    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.585576 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.585546    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"]
Nov 02 09:40:54.686542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.686507    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vhnd\" (UniqueName: \"kubernetes.io/projected/231615b6-94b8-440a-97ae-dcdca4860543-kube-api-access-5vhnd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.686689 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.686561    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231615b6-94b8-440a-97ae-dcdca4860543-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.686689 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.686638    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/231615b6-94b8-440a-97ae-dcdca4860543-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.686689 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.686678    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/231615b6-94b8-440a-97ae-dcdca4860543-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.686818 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.686707    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/231615b6-94b8-440a-97ae-dcdca4860543-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.686818 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.686784    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/231615b6-94b8-440a-97ae-dcdca4860543-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.686818 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.686808    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/231615b6-94b8-440a-97ae-dcdca4860543-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.787981 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.787896    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231615b6-94b8-440a-97ae-dcdca4860543-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.787981 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.787949    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/231615b6-94b8-440a-97ae-dcdca4860543-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.788178 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.787987    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/231615b6-94b8-440a-97ae-dcdca4860543-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.788178 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.788024    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/231615b6-94b8-440a-97ae-dcdca4860543-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.788178 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.788064    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/231615b6-94b8-440a-97ae-dcdca4860543-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.788178 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.788088    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/231615b6-94b8-440a-97ae-dcdca4860543-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.788178 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.788126    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vhnd\" (UniqueName: \"kubernetes.io/projected/231615b6-94b8-440a-97ae-dcdca4860543-kube-api-access-5vhnd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.788821 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.788796    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/231615b6-94b8-440a-97ae-dcdca4860543-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.790471 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.790448    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/231615b6-94b8-440a-97ae-dcdca4860543-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.790601 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.790456    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/231615b6-94b8-440a-97ae-dcdca4860543-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.791246 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.791222    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/231615b6-94b8-440a-97ae-dcdca4860543-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.791325 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.791259    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/231615b6-94b8-440a-97ae-dcdca4860543-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.801117 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.801092    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vhnd\" (UniqueName: \"kubernetes.io/projected/231615b6-94b8-440a-97ae-dcdca4860543-kube-api-access-5vhnd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.801789 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.801770    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231615b6-94b8-440a-97ae-dcdca4860543-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6p6b2\" (UID: \"231615b6-94b8-440a-97ae-dcdca4860543\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:54.819083 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.819057    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-cfpwn"]
Nov 02 09:40:54.822583 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:40:54.822561    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b53ec3_a9e4_45d7_8aa4_59237abc2614.slice/crio-af7df87c0e153722f207d59134205c40981685f34769836227799ae8f74c3aba WatchSource:0}: Error finding container af7df87c0e153722f207d59134205c40981685f34769836227799ae8f74c3aba: Status 404 returned error can't find the container with id af7df87c0e153722f207d59134205c40981685f34769836227799ae8f74c3aba
Nov 02 09:40:54.839705 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.839651    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"]
Nov 02 09:40:54.841540 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:40:54.841519    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6a1a4483_017b_4cb2_84e4_910f30499440.slice/crio-c95577de6edc18b169710fb38c0f3e4dda495502b3dc43015d033218053c2e2e WatchSource:0}: Error finding container c95577de6edc18b169710fb38c0f3e4dda495502b3dc43015d033218053c2e2e: Status 404 returned error can't find the container with id c95577de6edc18b169710fb38c0f3e4dda495502b3dc43015d033218053c2e2e
Nov 02 09:40:54.887754 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:54.887730    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:55.167567 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:55.167541    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"]
Nov 02 09:40:55.169994 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:40:55.169965    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231615b6_94b8_440a_97ae_dcdca4860543.slice/crio-946d74a59729809d50f9065d076b851c2112e0e02cbb20f16b578de69e8b83cb WatchSource:0}: Error finding container 946d74a59729809d50f9065d076b851c2112e0e02cbb20f16b578de69e8b83cb: Status 404 returned error can't find the container with id 946d74a59729809d50f9065d076b851c2112e0e02cbb20f16b578de69e8b83cb
Nov 02 09:40:55.171995 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:55.171966    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:40:55.172096 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:55.172033    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:40:55.796051 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:55.796007    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk" event={"ID":"6a1a4483-017b-4cb2-84e4-910f30499440","Type":"ContainerStarted","Data":"c95577de6edc18b169710fb38c0f3e4dda495502b3dc43015d033218053c2e2e"}
Nov 02 09:40:55.797496 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:55.797447    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-cfpwn" event={"ID":"b0b53ec3-a9e4-45d7-8aa4-59237abc2614","Type":"ContainerStarted","Data":"af7df87c0e153722f207d59134205c40981685f34769836227799ae8f74c3aba"}
Nov 02 09:40:55.799779 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:55.799754    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2" event={"ID":"231615b6-94b8-440a-97ae-dcdca4860543","Type":"ContainerStarted","Data":"15c275e6d45c50100ffd06279b7afecdfda984819c7b9a3c2f5f652946d3142c"}
Nov 02 09:40:55.799893 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:55.799802    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2" event={"ID":"231615b6-94b8-440a-97ae-dcdca4860543","Type":"ContainerStarted","Data":"946d74a59729809d50f9065d076b851c2112e0e02cbb20f16b578de69e8b83cb"}
Nov 02 09:40:55.800058 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:55.800035    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:55.827670 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:55.827612    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2" podStartSLOduration=1.827594776 podStartE2EDuration="1.827594776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:55.825098646 +0000 UTC m=+336.657756727" watchObservedRunningTime="2026-11-02 09:40:55.827594776 +0000 UTC m=+336.660252857"
Nov 02 09:40:56.807502 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:56.807468    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6p6b2"
Nov 02 09:40:56.866690 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:56.866656    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"]
Nov 02 09:40:56.866900 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:56.866875    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl" podUID="892ac505-5776-4642-b6f9-bb776186e86a" containerName="discovery" containerID="cri-o://662b1ef12f732e699f432030092103623cc35d10a2cc0e6e8534bfb8d663777f" gracePeriod=30
Nov 02 09:40:57.141234 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.141200    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:40:57.211869 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.211832    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-istio-kubeconfig\") pod \"892ac505-5776-4642-b6f9-bb776186e86a\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") "
Nov 02 09:40:57.212039 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.211898    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892ac505-5776-4642-b6f9-bb776186e86a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/892ac505-5776-4642-b6f9-bb776186e86a-istio-csr-ca-configmap\") pod \"892ac505-5776-4642-b6f9-bb776186e86a\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") "
Nov 02 09:40:57.212039 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.211934    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-istio-csr-dns-cert\") pod \"892ac505-5776-4642-b6f9-bb776186e86a\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") "
Nov 02 09:40:57.212039 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.211979    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892ac505-5776-4642-b6f9-bb776186e86a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/892ac505-5776-4642-b6f9-bb776186e86a-local-certs\") pod \"892ac505-5776-4642-b6f9-bb776186e86a\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") "
Nov 02 09:40:57.212172 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.212039    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892ac505-5776-4642-b6f9-bb776186e86a-kube-api-access-b88qj\" (UniqueName: \"kubernetes.io/projected/892ac505-5776-4642-b6f9-bb776186e86a-kube-api-access-b88qj\") pod \"892ac505-5776-4642-b6f9-bb776186e86a\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") "
Nov 02 09:40:57.212172 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.212078    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-cacerts\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-cacerts\") pod \"892ac505-5776-4642-b6f9-bb776186e86a\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") "
Nov 02 09:40:57.212172 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.212110    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892ac505-5776-4642-b6f9-bb776186e86a-istio-token\" (UniqueName: \"kubernetes.io/projected/892ac505-5776-4642-b6f9-bb776186e86a-istio-token\") pod \"892ac505-5776-4642-b6f9-bb776186e86a\" (UID: \"892ac505-5776-4642-b6f9-bb776186e86a\") "
Nov 02 09:40:57.212674 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.212639    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892ac505-5776-4642-b6f9-bb776186e86a-istio-csr-ca-configmap" pod "892ac505-5776-4642-b6f9-bb776186e86a" (UID: "892ac505-5776-4642-b6f9-bb776186e86a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:40:57.214745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.214720    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892ac505-5776-4642-b6f9-bb776186e86a-local-certs" pod "892ac505-5776-4642-b6f9-bb776186e86a" (UID: "892ac505-5776-4642-b6f9-bb776186e86a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:40:57.216074 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.216048    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-istio-kubeconfig" pod "892ac505-5776-4642-b6f9-bb776186e86a" (UID: "892ac505-5776-4642-b6f9-bb776186e86a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:40:57.216296 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.216259    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-cacerts" pod "892ac505-5776-4642-b6f9-bb776186e86a" (UID: "892ac505-5776-4642-b6f9-bb776186e86a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:40:57.216419 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.216401    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-istio-csr-dns-cert" pod "892ac505-5776-4642-b6f9-bb776186e86a" (UID: "892ac505-5776-4642-b6f9-bb776186e86a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:40:57.216898 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.216859    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892ac505-5776-4642-b6f9-bb776186e86a-kube-api-access-b88qj" pod "892ac505-5776-4642-b6f9-bb776186e86a" (UID: "892ac505-5776-4642-b6f9-bb776186e86a"). InnerVolumeSpecName "kube-api-access-b88qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:40:57.226709 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.226678    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892ac505-5776-4642-b6f9-bb776186e86a-istio-token" pod "892ac505-5776-4642-b6f9-bb776186e86a" (UID: "892ac505-5776-4642-b6f9-bb776186e86a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:40:57.313924 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.313894    4614 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-istio-kubeconfig\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:40:57.314081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.313926    4614 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/892ac505-5776-4642-b6f9-bb776186e86a-istio-csr-ca-configmap\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:40:57.314081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.313944    4614 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-istio-csr-dns-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:40:57.314081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.313958    4614 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/892ac505-5776-4642-b6f9-bb776186e86a-local-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:40:57.314081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.313973    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b88qj\" (UniqueName: \"kubernetes.io/projected/892ac505-5776-4642-b6f9-bb776186e86a-kube-api-access-b88qj\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:40:57.314081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.313987    4614 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/892ac505-5776-4642-b6f9-bb776186e86a-cacerts\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:40:57.314081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.314001    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892ac505-5776-4642-b6f9-bb776186e86a-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:40:57.811676 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.811638    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk" event={"ID":"6a1a4483-017b-4cb2-84e4-910f30499440","Type":"ContainerStarted","Data":"a3a126bb5c586b3d294a5d3a5d3a7af6eb7b5464c451fbf54465ddd940f7f5a7"}
Nov 02 09:40:57.812171 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.811831    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"
Nov 02 09:40:57.813218 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.813189    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-cfpwn" event={"ID":"b0b53ec3-a9e4-45d7-8aa4-59237abc2614","Type":"ContainerStarted","Data":"d91544277fa4a62b339fedd1f1d48b6f03abf507fc5d9dcf9768dc807bcf1495"}
Nov 02 09:40:57.813347 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.813300    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-cfpwn"
Nov 02 09:40:57.814406 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.814383    4614 generic.go:356] "Generic (PLEG): container finished" podID="892ac505-5776-4642-b6f9-bb776186e86a" containerID="662b1ef12f732e699f432030092103623cc35d10a2cc0e6e8534bfb8d663777f" exitCode=0
Nov 02 09:40:57.814469 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.814438    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"
Nov 02 09:40:57.814512 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.814465    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl" event={"ID":"892ac505-5776-4642-b6f9-bb776186e86a","Type":"ContainerDied","Data":"662b1ef12f732e699f432030092103623cc35d10a2cc0e6e8534bfb8d663777f"}
Nov 02 09:40:57.814512 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.814506    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl" event={"ID":"892ac505-5776-4642-b6f9-bb776186e86a","Type":"ContainerDied","Data":"be53dddc04a8c147e306dd24213d3ce34500557e8f2d3ca34b8819d889edabac"}
Nov 02 09:40:57.814580 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.814529    4614 scope.go:122] "RemoveContainer" containerID="662b1ef12f732e699f432030092103623cc35d10a2cc0e6e8534bfb8d663777f"
Nov 02 09:40:57.822685 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.822668    4614 scope.go:122] "RemoveContainer" containerID="662b1ef12f732e699f432030092103623cc35d10a2cc0e6e8534bfb8d663777f"
Nov 02 09:40:57.822934 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:40:57.822918    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662b1ef12f732e699f432030092103623cc35d10a2cc0e6e8534bfb8d663777f\": container with ID starting with 662b1ef12f732e699f432030092103623cc35d10a2cc0e6e8534bfb8d663777f not found: ID does not exist" containerID="662b1ef12f732e699f432030092103623cc35d10a2cc0e6e8534bfb8d663777f"
Nov 02 09:40:57.822986 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.822940    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662b1ef12f732e699f432030092103623cc35d10a2cc0e6e8534bfb8d663777f"} err="failed to get container status \"662b1ef12f732e699f432030092103623cc35d10a2cc0e6e8534bfb8d663777f\": rpc error: code = NotFound desc = could not find container \"662b1ef12f732e699f432030092103623cc35d10a2cc0e6e8534bfb8d663777f\": container with ID starting with 662b1ef12f732e699f432030092103623cc35d10a2cc0e6e8534bfb8d663777f not found: ID does not exist"
Nov 02 09:40:57.830332 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.830289    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk" podStartSLOduration=1.075846574 podStartE2EDuration="3.830277231s" totalImagesPullingTime="2.754430657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:57.830107321 +0000 UTC m=+338.662765402" watchObservedRunningTime="2026-11-02 09:40:57.830277231 +0000 UTC m=+338.662935307"
Nov 02 09:40:57.850464 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.850418    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-cfpwn" podStartSLOduration=1.326276018 podStartE2EDuration="3.850402682s" totalImagesPullingTime="2.524126664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:40:57.848236276 +0000 UTC m=+338.680894359" watchObservedRunningTime="2026-11-02 09:40:57.850402682 +0000 UTC m=+338.683060764"
Nov 02 09:40:57.862166 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.862142    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"]
Nov 02 09:40:57.867282 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:57.867260    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4dlgl"]
Nov 02 09:40:59.314770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:59.314735    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892ac505-5776-4642-b6f9-bb776186e86a" path="/var/lib/kubelet/pods/892ac505-5776-4642-b6f9-bb776186e86a/volumes"
Nov 02 09:40:59.816153 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:40:59.816127    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-sd2jc"
Nov 02 09:41:07.369232 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.369191    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-796596f549-zjfq7"]
Nov 02 09:41:07.369783 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.369551    4614 cpu_manager.go:414] "containerMap: removing container" podUID="892ac505-5776-4642-b6f9-bb776186e86a" containerName="discovery"
Nov 02 09:41:07.369783 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.369562    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892ac505-5776-4642-b6f9-bb776186e86a" containerName="discovery"
Nov 02 09:41:07.369783 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.369643    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="892ac505-5776-4642-b6f9-bb776186e86a" containerName="discovery"
Nov 02 09:41:07.372806 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.372789    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:07.377446 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.377414    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 09:41:07.377446 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.377431    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zrg5v\""
Nov 02 09:41:07.377446 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.377423    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 09:41:07.377658 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.377550    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 09:41:07.384190 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.384163    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-796596f549-zjfq7"]
Nov 02 09:41:07.496623 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.496582    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48dgb\" (UniqueName: \"kubernetes.io/projected/a3ec11dd-9317-4377-86de-8ee0a85b4685-kube-api-access-48dgb\") pod \"maas-controller-796596f549-zjfq7\" (UID: \"a3ec11dd-9317-4377-86de-8ee0a85b4685\") " pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:07.496803 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.496699    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3ec11dd-9317-4377-86de-8ee0a85b4685-webhook-cert\") pod \"maas-controller-796596f549-zjfq7\" (UID: \"a3ec11dd-9317-4377-86de-8ee0a85b4685\") " pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:07.496803 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.496783    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ec11dd-9317-4377-86de-8ee0a85b4685-metrics-certs\") pod \"maas-controller-796596f549-zjfq7\" (UID: \"a3ec11dd-9317-4377-86de-8ee0a85b4685\") " pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:07.597419 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.597380    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3ec11dd-9317-4377-86de-8ee0a85b4685-webhook-cert\") pod \"maas-controller-796596f549-zjfq7\" (UID: \"a3ec11dd-9317-4377-86de-8ee0a85b4685\") " pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:07.597419 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.597428    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ec11dd-9317-4377-86de-8ee0a85b4685-metrics-certs\") pod \"maas-controller-796596f549-zjfq7\" (UID: \"a3ec11dd-9317-4377-86de-8ee0a85b4685\") " pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:07.597707 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.597479    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48dgb\" (UniqueName: \"kubernetes.io/projected/a3ec11dd-9317-4377-86de-8ee0a85b4685-kube-api-access-48dgb\") pod \"maas-controller-796596f549-zjfq7\" (UID: \"a3ec11dd-9317-4377-86de-8ee0a85b4685\") " pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:07.599993 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.599959    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3ec11dd-9317-4377-86de-8ee0a85b4685-webhook-cert\") pod \"maas-controller-796596f549-zjfq7\" (UID: \"a3ec11dd-9317-4377-86de-8ee0a85b4685\") " pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:07.600112 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.599992    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ec11dd-9317-4377-86de-8ee0a85b4685-metrics-certs\") pod \"maas-controller-796596f549-zjfq7\" (UID: \"a3ec11dd-9317-4377-86de-8ee0a85b4685\") " pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:07.609675 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.609648    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48dgb\" (UniqueName: \"kubernetes.io/projected/a3ec11dd-9317-4377-86de-8ee0a85b4685-kube-api-access-48dgb\") pod \"maas-controller-796596f549-zjfq7\" (UID: \"a3ec11dd-9317-4377-86de-8ee0a85b4685\") " pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:07.691753 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.691666    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:07.909407 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.909374    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b555b5c85-jv2xg"]
Nov 02 09:41:07.913052 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.913035    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 09:41:07.923870 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.923846    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b555b5c85-jv2xg"]
Nov 02 09:41:07.966892 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.966823    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-796596f549-zjfq7"]
Nov 02 09:41:07.969916 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:41:07.969891    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ec11dd_9317_4377_86de_8ee0a85b4685.slice/crio-747bd94b755c8d8fc883ac16e94287750550b9a8098306d5a6f02454e2575529 WatchSource:0}: Error finding container 747bd94b755c8d8fc883ac16e94287750550b9a8098306d5a6f02454e2575529: Status 404 returned error can't find the container with id 747bd94b755c8d8fc883ac16e94287750550b9a8098306d5a6f02454e2575529
Nov 02 09:41:07.999720 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.999694    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stf5z\" (UniqueName: \"kubernetes.io/projected/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-kube-api-access-stf5z\") pod \"maas-controller-5b555b5c85-jv2xg\" (UID: \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\") " pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 09:41:07.999824 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.999736    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-webhook-cert\") pod \"maas-controller-5b555b5c85-jv2xg\" (UID: \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\") " pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 09:41:07.999875 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:07.999834    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-metrics-certs\") pod \"maas-controller-5b555b5c85-jv2xg\" (UID: \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\") " pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 09:41:08.100691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:08.100658    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-metrics-certs\") pod \"maas-controller-5b555b5c85-jv2xg\" (UID: \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\") " pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 09:41:08.100858 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:08.100726    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stf5z\" (UniqueName: \"kubernetes.io/projected/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-kube-api-access-stf5z\") pod \"maas-controller-5b555b5c85-jv2xg\" (UID: \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\") " pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 09:41:08.100858 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:08.100782    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-webhook-cert\") pod \"maas-controller-5b555b5c85-jv2xg\" (UID: \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\") " pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 09:41:08.103213 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:08.103192    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-webhook-cert\") pod \"maas-controller-5b555b5c85-jv2xg\" (UID: \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\") " pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 09:41:08.103309 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:08.103217    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-metrics-certs\") pod \"maas-controller-5b555b5c85-jv2xg\" (UID: \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\") " pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 09:41:08.114004 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:08.113983    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stf5z\" (UniqueName: \"kubernetes.io/projected/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-kube-api-access-stf5z\") pod \"maas-controller-5b555b5c85-jv2xg\" (UID: \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\") " pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 09:41:08.227727 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:08.227697    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 09:41:08.502376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:08.502236    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b555b5c85-jv2xg"]
Nov 02 09:41:08.504828 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:41:08.504802    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fcb0a2_7557_4805_aff1_ac13b3ba44e4.slice/crio-7a3921c42f2d5cd8aebd1bc11e4a78d8ec3f13d98df319ac1174111fa08ee355 WatchSource:0}: Error finding container 7a3921c42f2d5cd8aebd1bc11e4a78d8ec3f13d98df319ac1174111fa08ee355: Status 404 returned error can't find the container with id 7a3921c42f2d5cd8aebd1bc11e4a78d8ec3f13d98df319ac1174111fa08ee355
Nov 02 09:41:08.868069 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:08.867992    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b555b5c85-jv2xg" event={"ID":"43fcb0a2-7557-4805-aff1-ac13b3ba44e4","Type":"ContainerStarted","Data":"7a3921c42f2d5cd8aebd1bc11e4a78d8ec3f13d98df319ac1174111fa08ee355"}
Nov 02 09:41:08.868997 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:08.868975    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796596f549-zjfq7" event={"ID":"a3ec11dd-9317-4377-86de-8ee0a85b4685","Type":"ContainerStarted","Data":"747bd94b755c8d8fc883ac16e94287750550b9a8098306d5a6f02454e2575529"}
Nov 02 09:41:09.640846 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:41:09.640806    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:41:12.892000 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:12.891952    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b555b5c85-jv2xg" event={"ID":"43fcb0a2-7557-4805-aff1-ac13b3ba44e4","Type":"ContainerStarted","Data":"04d9524770da0d23925a2dfecc8009167d7580195bd683c1dd84a3a6b0e6a223"}
Nov 02 09:41:12.892505 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:12.892052    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 09:41:12.893622 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:12.893588    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796596f549-zjfq7" event={"ID":"a3ec11dd-9317-4377-86de-8ee0a85b4685","Type":"ContainerStarted","Data":"2cc6f1b01e98342f0f1788728a67a2601ec80064063a78bd5bba21d87a799d18"}
Nov 02 09:41:12.893742 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:12.893729    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:12.914213 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:12.914159    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5b555b5c85-jv2xg" podStartSLOduration=2.4844106249999998 podStartE2EDuration="5.91414599s" totalImagesPullingTime="3.429735365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:41:12.913477542 +0000 UTC m=+353.746135633" watchObservedRunningTime="2026-11-02 09:41:12.91414599 +0000 UTC m=+353.746804070"
Nov 02 09:41:12.936721 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:12.936674    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-796596f549-zjfq7" podStartSLOduration=1.98063258 podStartE2EDuration="5.936654386s" totalImagesPullingTime="3.956021806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:41:12.93524373 +0000 UTC m=+353.767901810" watchObservedRunningTime="2026-11-02 09:41:12.936654386 +0000 UTC m=+353.769312487"
Nov 02 09:41:23.900592 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:23.900555    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 09:41:23.901043 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:23.900730    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:23.975576 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:23.975540    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-796596f549-zjfq7"]
Nov 02 09:41:23.975773 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:23.975751    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-796596f549-zjfq7" podUID="a3ec11dd-9317-4377-86de-8ee0a85b4685" containerName="manager" containerID="cri-o://2cc6f1b01e98342f0f1788728a67a2601ec80064063a78bd5bba21d87a799d18" gracePeriod=30
Nov 02 09:41:24.216450 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.216426    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:24.235104 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.235078    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3ec11dd-9317-4377-86de-8ee0a85b4685-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3ec11dd-9317-4377-86de-8ee0a85b4685-webhook-cert\") pod \"a3ec11dd-9317-4377-86de-8ee0a85b4685\" (UID: \"a3ec11dd-9317-4377-86de-8ee0a85b4685\") "
Nov 02 09:41:24.235218 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.235167    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ec11dd-9317-4377-86de-8ee0a85b4685-kube-api-access-48dgb\" (UniqueName: \"kubernetes.io/projected/a3ec11dd-9317-4377-86de-8ee0a85b4685-kube-api-access-48dgb\") pod \"a3ec11dd-9317-4377-86de-8ee0a85b4685\" (UID: \"a3ec11dd-9317-4377-86de-8ee0a85b4685\") "
Nov 02 09:41:24.235218 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.235200    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3ec11dd-9317-4377-86de-8ee0a85b4685-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ec11dd-9317-4377-86de-8ee0a85b4685-metrics-certs\") pod \"a3ec11dd-9317-4377-86de-8ee0a85b4685\" (UID: \"a3ec11dd-9317-4377-86de-8ee0a85b4685\") "
Nov 02 09:41:24.237271 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.237237    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ec11dd-9317-4377-86de-8ee0a85b4685-metrics-certs" pod "a3ec11dd-9317-4377-86de-8ee0a85b4685" (UID: "a3ec11dd-9317-4377-86de-8ee0a85b4685"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:41:24.237420 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.237254    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ec11dd-9317-4377-86de-8ee0a85b4685-webhook-cert" pod "a3ec11dd-9317-4377-86de-8ee0a85b4685" (UID: "a3ec11dd-9317-4377-86de-8ee0a85b4685"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:41:24.237420 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.237332    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ec11dd-9317-4377-86de-8ee0a85b4685-kube-api-access-48dgb" pod "a3ec11dd-9317-4377-86de-8ee0a85b4685" (UID: "a3ec11dd-9317-4377-86de-8ee0a85b4685"). InnerVolumeSpecName "kube-api-access-48dgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:41:24.336668 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.336625    4614 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3ec11dd-9317-4377-86de-8ee0a85b4685-webhook-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:41:24.336668 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.336666    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48dgb\" (UniqueName: \"kubernetes.io/projected/a3ec11dd-9317-4377-86de-8ee0a85b4685-kube-api-access-48dgb\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:41:24.336668 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.336678    4614 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ec11dd-9317-4377-86de-8ee0a85b4685-metrics-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:41:24.948775 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.948740    4614 generic.go:356] "Generic (PLEG): container finished" podID="a3ec11dd-9317-4377-86de-8ee0a85b4685" containerID="2cc6f1b01e98342f0f1788728a67a2601ec80064063a78bd5bba21d87a799d18" exitCode=0
Nov 02 09:41:24.949191 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.948786    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796596f549-zjfq7"
Nov 02 09:41:24.949191 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.948833    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796596f549-zjfq7" event={"ID":"a3ec11dd-9317-4377-86de-8ee0a85b4685","Type":"ContainerDied","Data":"2cc6f1b01e98342f0f1788728a67a2601ec80064063a78bd5bba21d87a799d18"}
Nov 02 09:41:24.949191 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.948883    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796596f549-zjfq7" event={"ID":"a3ec11dd-9317-4377-86de-8ee0a85b4685","Type":"ContainerDied","Data":"747bd94b755c8d8fc883ac16e94287750550b9a8098306d5a6f02454e2575529"}
Nov 02 09:41:24.949191 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.948906    4614 scope.go:122] "RemoveContainer" containerID="2cc6f1b01e98342f0f1788728a67a2601ec80064063a78bd5bba21d87a799d18"
Nov 02 09:41:24.957556 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.957540    4614 scope.go:122] "RemoveContainer" containerID="2cc6f1b01e98342f0f1788728a67a2601ec80064063a78bd5bba21d87a799d18"
Nov 02 09:41:24.957827 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:41:24.957810    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc6f1b01e98342f0f1788728a67a2601ec80064063a78bd5bba21d87a799d18\": container with ID starting with 2cc6f1b01e98342f0f1788728a67a2601ec80064063a78bd5bba21d87a799d18 not found: ID does not exist" containerID="2cc6f1b01e98342f0f1788728a67a2601ec80064063a78bd5bba21d87a799d18"
Nov 02 09:41:24.957870 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.957840    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc6f1b01e98342f0f1788728a67a2601ec80064063a78bd5bba21d87a799d18"} err="failed to get container status \"2cc6f1b01e98342f0f1788728a67a2601ec80064063a78bd5bba21d87a799d18\": rpc error: code = NotFound desc = could not find container \"2cc6f1b01e98342f0f1788728a67a2601ec80064063a78bd5bba21d87a799d18\": container with ID starting with 2cc6f1b01e98342f0f1788728a67a2601ec80064063a78bd5bba21d87a799d18 not found: ID does not exist"
Nov 02 09:41:24.970432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.970410    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-796596f549-zjfq7"]
Nov 02 09:41:24.974395 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:24.974375    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-796596f549-zjfq7"]
Nov 02 09:41:25.314729 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:25.314698    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ec11dd-9317-4377-86de-8ee0a85b4685" path="/var/lib/kubelet/pods/a3ec11dd-9317-4377-86de-8ee0a85b4685/volumes"
Nov 02 09:41:28.822160 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:28.822128    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"
Nov 02 09:41:28.825137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:28.825119    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-cfpwn"
Nov 02 09:41:39.636035 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:41:39.635988    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:41:45.670235 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.670194    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"]
Nov 02 09:41:45.670738 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.670693    4614 cpu_manager.go:414] "containerMap: removing container" podUID="a3ec11dd-9317-4377-86de-8ee0a85b4685" containerName="manager"
Nov 02 09:41:45.670738 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.670713    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ec11dd-9317-4377-86de-8ee0a85b4685" containerName="manager"
Nov 02 09:41:45.670857 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.670846    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ec11dd-9317-4377-86de-8ee0a85b4685" containerName="manager"
Nov 02 09:41:45.677745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.677722    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:45.685405 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.685367    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 09:41:45.685405 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.685393    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5thw\""
Nov 02 09:41:45.685624 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.685593    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 09:41:45.687056 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.687039    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"]
Nov 02 09:41:45.821687 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.821653    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eba26412-4424-49e6-9cf2-0117a0c1ff80-metrics-certs\") pod \"maas-api-74b6f69d8-kvkdf\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") " pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:45.821687 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.821686    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba26412-4424-49e6-9cf2-0117a0c1ff80-openshift-service-ca\") pod \"maas-api-74b6f69d8-kvkdf\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") " pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:45.821900 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.821808    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5q66\" (UniqueName: \"kubernetes.io/projected/eba26412-4424-49e6-9cf2-0117a0c1ff80-kube-api-access-w5q66\") pod \"maas-api-74b6f69d8-kvkdf\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") " pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:45.821900 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.821860    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eba26412-4424-49e6-9cf2-0117a0c1ff80-maas-api-tls\") pod \"maas-api-74b6f69d8-kvkdf\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") " pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:45.923026 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.922941    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5q66\" (UniqueName: \"kubernetes.io/projected/eba26412-4424-49e6-9cf2-0117a0c1ff80-kube-api-access-w5q66\") pod \"maas-api-74b6f69d8-kvkdf\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") " pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:45.923026 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.922998    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eba26412-4424-49e6-9cf2-0117a0c1ff80-maas-api-tls\") pod \"maas-api-74b6f69d8-kvkdf\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") " pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:45.923026 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.923025    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eba26412-4424-49e6-9cf2-0117a0c1ff80-metrics-certs\") pod \"maas-api-74b6f69d8-kvkdf\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") " pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:45.923308 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.923041    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba26412-4424-49e6-9cf2-0117a0c1ff80-openshift-service-ca\") pod \"maas-api-74b6f69d8-kvkdf\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") " pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:45.923674 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.923647    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba26412-4424-49e6-9cf2-0117a0c1ff80-openshift-service-ca\") pod \"maas-api-74b6f69d8-kvkdf\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") " pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:45.925631 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.925595    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eba26412-4424-49e6-9cf2-0117a0c1ff80-metrics-certs\") pod \"maas-api-74b6f69d8-kvkdf\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") " pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:45.925841 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.925822    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eba26412-4424-49e6-9cf2-0117a0c1ff80-maas-api-tls\") pod \"maas-api-74b6f69d8-kvkdf\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") " pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:45.935447 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.935422    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5q66\" (UniqueName: \"kubernetes.io/projected/eba26412-4424-49e6-9cf2-0117a0c1ff80-kube-api-access-w5q66\") pod \"maas-api-74b6f69d8-kvkdf\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") " pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:45.991200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:45.991167    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:46.267585 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:46.267525    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"]
Nov 02 09:41:46.269880 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:41:46.269852    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba26412_4424_49e6_9cf2_0117a0c1ff80.slice/crio-4f7f10fbe2a35dff8fd718ed0079ca40038785fc3b574471f1a0e73f0403bb39 WatchSource:0}: Error finding container 4f7f10fbe2a35dff8fd718ed0079ca40038785fc3b574471f1a0e73f0403bb39: Status 404 returned error can't find the container with id 4f7f10fbe2a35dff8fd718ed0079ca40038785fc3b574471f1a0e73f0403bb39
Nov 02 09:41:47.043441 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:47.043405    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf" event={"ID":"eba26412-4424-49e6-9cf2-0117a0c1ff80","Type":"ContainerStarted","Data":"4f7f10fbe2a35dff8fd718ed0079ca40038785fc3b574471f1a0e73f0403bb39"}
Nov 02 09:41:50.057574 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:50.057538    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf" event={"ID":"eba26412-4424-49e6-9cf2-0117a0c1ff80","Type":"ContainerStarted","Data":"9e5e5da680b84cdd1c95746d9d0eb11777867807b007d2fa6afaf23cb1481b43"}
Nov 02 09:41:50.057988 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:50.057638    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:41:50.078430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:50.078384    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf" podStartSLOduration=1.800587358 podStartE2EDuration="5.078347224s" totalImagesPullingTime="3.277759866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:41:50.076970295 +0000 UTC m=+390.909628375" watchObservedRunningTime="2026-11-02 09:41:50.078347224 +0000 UTC m=+390.911005304"
Nov 02 09:41:56.067345 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:41:56.067317    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:42:00.327237 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:00.327160    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd"]
Nov 02 09:42:00.331906 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:00.331884    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd"
Nov 02 09:42:00.342331 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:00.342303    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd"]
Nov 02 09:42:00.443649 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:00.443611    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkr5d\" (UniqueName: \"kubernetes.io/projected/7f946e8c-3ccb-4967-9d2d-61f67abb41ed-kube-api-access-wkr5d\") pod \"llmisvc-controller-manager-6466c554d8-j8qxd\" (UID: \"7f946e8c-3ccb-4967-9d2d-61f67abb41ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd"
Nov 02 09:42:00.443823 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:00.443661    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f946e8c-3ccb-4967-9d2d-61f67abb41ed-cert\") pod \"llmisvc-controller-manager-6466c554d8-j8qxd\" (UID: \"7f946e8c-3ccb-4967-9d2d-61f67abb41ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd"
Nov 02 09:42:00.544795 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:00.544761    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkr5d\" (UniqueName: \"kubernetes.io/projected/7f946e8c-3ccb-4967-9d2d-61f67abb41ed-kube-api-access-wkr5d\") pod \"llmisvc-controller-manager-6466c554d8-j8qxd\" (UID: \"7f946e8c-3ccb-4967-9d2d-61f67abb41ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd"
Nov 02 09:42:00.544958 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:00.544806    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f946e8c-3ccb-4967-9d2d-61f67abb41ed-cert\") pod \"llmisvc-controller-manager-6466c554d8-j8qxd\" (UID: \"7f946e8c-3ccb-4967-9d2d-61f67abb41ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd"
Nov 02 09:42:00.547267 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:00.547240    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f946e8c-3ccb-4967-9d2d-61f67abb41ed-cert\") pod \"llmisvc-controller-manager-6466c554d8-j8qxd\" (UID: \"7f946e8c-3ccb-4967-9d2d-61f67abb41ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd"
Nov 02 09:42:00.557485 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:00.557461    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkr5d\" (UniqueName: \"kubernetes.io/projected/7f946e8c-3ccb-4967-9d2d-61f67abb41ed-kube-api-access-wkr5d\") pod \"llmisvc-controller-manager-6466c554d8-j8qxd\" (UID: \"7f946e8c-3ccb-4967-9d2d-61f67abb41ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd"
Nov 02 09:42:00.645196 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:00.645123    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd"
Nov 02 09:42:00.921068 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:00.921037    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd"]
Nov 02 09:42:00.923425 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:42:00.923393    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f946e8c_3ccb_4967_9d2d_61f67abb41ed.slice/crio-34ebc0255f4c92855d400c3009175252bfb7fb1afd73654ade624fc0d57929d8 WatchSource:0}: Error finding container 34ebc0255f4c92855d400c3009175252bfb7fb1afd73654ade624fc0d57929d8: Status 404 returned error can't find the container with id 34ebc0255f4c92855d400c3009175252bfb7fb1afd73654ade624fc0d57929d8
Nov 02 09:42:01.110285 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:01.110250    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd" event={"ID":"7f946e8c-3ccb-4967-9d2d-61f67abb41ed","Type":"ContainerStarted","Data":"34ebc0255f4c92855d400c3009175252bfb7fb1afd73654ade624fc0d57929d8"}
Nov 02 09:42:02.116673 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:02.116634    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd" event={"ID":"7f946e8c-3ccb-4967-9d2d-61f67abb41ed","Type":"ContainerStarted","Data":"b5106a3a896bd0ef85998f9e4020732e66cd8129dcbca99746a84401cbd54fb9"}
Nov 02 09:42:02.117089 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:02.116756    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd"
Nov 02 09:42:02.140410 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:02.140347    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd" podStartSLOduration=1.756865995 podStartE2EDuration="2.140333401s" totalImagesPullingTime="383.467406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:42:02.139069864 +0000 UTC m=+402.971727943" watchObservedRunningTime="2026-11-02 09:42:02.140333401 +0000 UTC m=+402.972991480"
Nov 02 09:42:07.152571 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.152530    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"]
Nov 02 09:42:07.153059 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.152830    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf" podUID="eba26412-4424-49e6-9cf2-0117a0c1ff80" containerName="maas-api" containerID="cri-o://9e5e5da680b84cdd1c95746d9d0eb11777867807b007d2fa6afaf23cb1481b43" gracePeriod=30
Nov 02 09:42:07.408974 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.408913    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:42:07.498577 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.498539    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eba26412-4424-49e6-9cf2-0117a0c1ff80-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eba26412-4424-49e6-9cf2-0117a0c1ff80-metrics-certs\") pod \"eba26412-4424-49e6-9cf2-0117a0c1ff80\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") "
Nov 02 09:42:07.498751 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.498594    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eba26412-4424-49e6-9cf2-0117a0c1ff80-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba26412-4424-49e6-9cf2-0117a0c1ff80-openshift-service-ca\") pod \"eba26412-4424-49e6-9cf2-0117a0c1ff80\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") "
Nov 02 09:42:07.498751 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.498627    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba26412-4424-49e6-9cf2-0117a0c1ff80-kube-api-access-w5q66\" (UniqueName: \"kubernetes.io/projected/eba26412-4424-49e6-9cf2-0117a0c1ff80-kube-api-access-w5q66\") pod \"eba26412-4424-49e6-9cf2-0117a0c1ff80\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") "
Nov 02 09:42:07.498859 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.498812    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eba26412-4424-49e6-9cf2-0117a0c1ff80-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eba26412-4424-49e6-9cf2-0117a0c1ff80-maas-api-tls\") pod \"eba26412-4424-49e6-9cf2-0117a0c1ff80\" (UID: \"eba26412-4424-49e6-9cf2-0117a0c1ff80\") "
Nov 02 09:42:07.499066 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.499032    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba26412-4424-49e6-9cf2-0117a0c1ff80-openshift-service-ca" pod "eba26412-4424-49e6-9cf2-0117a0c1ff80" (UID: "eba26412-4424-49e6-9cf2-0117a0c1ff80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:42:07.501016 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.500985    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba26412-4424-49e6-9cf2-0117a0c1ff80-maas-api-tls" pod "eba26412-4424-49e6-9cf2-0117a0c1ff80" (UID: "eba26412-4424-49e6-9cf2-0117a0c1ff80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:42:07.501141 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.501028    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba26412-4424-49e6-9cf2-0117a0c1ff80-kube-api-access-w5q66" pod "eba26412-4424-49e6-9cf2-0117a0c1ff80" (UID: "eba26412-4424-49e6-9cf2-0117a0c1ff80"). InnerVolumeSpecName "kube-api-access-w5q66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:42:07.501141 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.501078    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba26412-4424-49e6-9cf2-0117a0c1ff80-metrics-certs" pod "eba26412-4424-49e6-9cf2-0117a0c1ff80" (UID: "eba26412-4424-49e6-9cf2-0117a0c1ff80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:42:07.600108 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.600073    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eba26412-4424-49e6-9cf2-0117a0c1ff80-maas-api-tls\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:42:07.600108 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.600103    4614 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eba26412-4424-49e6-9cf2-0117a0c1ff80-metrics-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:42:07.600108 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.600115    4614 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba26412-4424-49e6-9cf2-0117a0c1ff80-openshift-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:42:07.600344 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:07.600127    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5q66\" (UniqueName: \"kubernetes.io/projected/eba26412-4424-49e6-9cf2-0117a0c1ff80-kube-api-access-w5q66\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:42:08.144188 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:08.144146    4614 generic.go:356] "Generic (PLEG): container finished" podID="eba26412-4424-49e6-9cf2-0117a0c1ff80" containerID="9e5e5da680b84cdd1c95746d9d0eb11777867807b007d2fa6afaf23cb1481b43" exitCode=0
Nov 02 09:42:08.144392 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:08.144195    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"
Nov 02 09:42:08.144392 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:08.144199    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf" event={"ID":"eba26412-4424-49e6-9cf2-0117a0c1ff80","Type":"ContainerDied","Data":"9e5e5da680b84cdd1c95746d9d0eb11777867807b007d2fa6afaf23cb1481b43"}
Nov 02 09:42:08.144392 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:08.144327    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf" event={"ID":"eba26412-4424-49e6-9cf2-0117a0c1ff80","Type":"ContainerDied","Data":"4f7f10fbe2a35dff8fd718ed0079ca40038785fc3b574471f1a0e73f0403bb39"}
Nov 02 09:42:08.144392 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:08.144366    4614 scope.go:122] "RemoveContainer" containerID="9e5e5da680b84cdd1c95746d9d0eb11777867807b007d2fa6afaf23cb1481b43"
Nov 02 09:42:08.154094 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:08.153843    4614 scope.go:122] "RemoveContainer" containerID="9e5e5da680b84cdd1c95746d9d0eb11777867807b007d2fa6afaf23cb1481b43"
Nov 02 09:42:08.154331 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:42:08.154138    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5e5da680b84cdd1c95746d9d0eb11777867807b007d2fa6afaf23cb1481b43\": container with ID starting with 9e5e5da680b84cdd1c95746d9d0eb11777867807b007d2fa6afaf23cb1481b43 not found: ID does not exist" containerID="9e5e5da680b84cdd1c95746d9d0eb11777867807b007d2fa6afaf23cb1481b43"
Nov 02 09:42:08.154331 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:08.154164    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5e5da680b84cdd1c95746d9d0eb11777867807b007d2fa6afaf23cb1481b43"} err="failed to get container status \"9e5e5da680b84cdd1c95746d9d0eb11777867807b007d2fa6afaf23cb1481b43\": rpc error: code = NotFound desc = could not find container \"9e5e5da680b84cdd1c95746d9d0eb11777867807b007d2fa6afaf23cb1481b43\": container with ID starting with 9e5e5da680b84cdd1c95746d9d0eb11777867807b007d2fa6afaf23cb1481b43 not found: ID does not exist"
Nov 02 09:42:08.167291 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:08.167262    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"]
Nov 02 09:42:08.171166 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:08.171142    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-74b6f69d8-kvkdf"]
Nov 02 09:42:09.314316 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:09.314280    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba26412-4424-49e6-9cf2-0117a0c1ff80" path="/var/lib/kubelet/pods/eba26412-4424-49e6-9cf2-0117a0c1ff80/volumes"
Nov 02 09:42:09.634947 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:42:09.634870    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:42:33.123488 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:33.123459    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j8qxd"
Nov 02 09:42:33.173432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:33.173394    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"]
Nov 02 09:42:33.173662 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:33.173602    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk" podUID="6a1a4483-017b-4cb2-84e4-910f30499440" containerName="manager" containerID="cri-o://a3a126bb5c586b3d294a5d3a5d3a7af6eb7b5464c451fbf54465ddd940f7f5a7" gracePeriod=30
Nov 02 09:42:33.414478 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:33.414454    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"
Nov 02 09:42:33.490568 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:33.490534    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1a4483-017b-4cb2-84e4-910f30499440-kube-api-access-lpgkg\" (UniqueName: \"kubernetes.io/projected/6a1a4483-017b-4cb2-84e4-910f30499440-kube-api-access-lpgkg\") pod \"6a1a4483-017b-4cb2-84e4-910f30499440\" (UID: \"6a1a4483-017b-4cb2-84e4-910f30499440\") "
Nov 02 09:42:33.490754 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:33.490660    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1a4483-017b-4cb2-84e4-910f30499440-cert\" (UniqueName: \"kubernetes.io/secret/6a1a4483-017b-4cb2-84e4-910f30499440-cert\") pod \"6a1a4483-017b-4cb2-84e4-910f30499440\" (UID: \"6a1a4483-017b-4cb2-84e4-910f30499440\") "
Nov 02 09:42:33.492585 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:33.492554    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1a4483-017b-4cb2-84e4-910f30499440-cert" pod "6a1a4483-017b-4cb2-84e4-910f30499440" (UID: "6a1a4483-017b-4cb2-84e4-910f30499440"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:42:33.492691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:33.492583    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1a4483-017b-4cb2-84e4-910f30499440-kube-api-access-lpgkg" pod "6a1a4483-017b-4cb2-84e4-910f30499440" (UID: "6a1a4483-017b-4cb2-84e4-910f30499440"). InnerVolumeSpecName "kube-api-access-lpgkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:42:33.591330 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:33.591299    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpgkg\" (UniqueName: \"kubernetes.io/projected/6a1a4483-017b-4cb2-84e4-910f30499440-kube-api-access-lpgkg\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:42:33.591330 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:33.591324    4614 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a1a4483-017b-4cb2-84e4-910f30499440-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:42:34.257417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:34.257382    4614 generic.go:356] "Generic (PLEG): container finished" podID="6a1a4483-017b-4cb2-84e4-910f30499440" containerID="a3a126bb5c586b3d294a5d3a5d3a7af6eb7b5464c451fbf54465ddd940f7f5a7" exitCode=0
Nov 02 09:42:34.257866 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:34.257435    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"
Nov 02 09:42:34.257866 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:34.257434    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk" event={"ID":"6a1a4483-017b-4cb2-84e4-910f30499440","Type":"ContainerDied","Data":"a3a126bb5c586b3d294a5d3a5d3a7af6eb7b5464c451fbf54465ddd940f7f5a7"}
Nov 02 09:42:34.257866 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:34.257542    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk" event={"ID":"6a1a4483-017b-4cb2-84e4-910f30499440","Type":"ContainerDied","Data":"c95577de6edc18b169710fb38c0f3e4dda495502b3dc43015d033218053c2e2e"}
Nov 02 09:42:34.257866 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:34.257571    4614 scope.go:122] "RemoveContainer" containerID="a3a126bb5c586b3d294a5d3a5d3a7af6eb7b5464c451fbf54465ddd940f7f5a7"
Nov 02 09:42:34.269782 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:34.269762    4614 scope.go:122] "RemoveContainer" containerID="a3a126bb5c586b3d294a5d3a5d3a7af6eb7b5464c451fbf54465ddd940f7f5a7"
Nov 02 09:42:34.270061 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:42:34.270041    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a126bb5c586b3d294a5d3a5d3a7af6eb7b5464c451fbf54465ddd940f7f5a7\": container with ID starting with a3a126bb5c586b3d294a5d3a5d3a7af6eb7b5464c451fbf54465ddd940f7f5a7 not found: ID does not exist" containerID="a3a126bb5c586b3d294a5d3a5d3a7af6eb7b5464c451fbf54465ddd940f7f5a7"
Nov 02 09:42:34.270125 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:34.270074    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a126bb5c586b3d294a5d3a5d3a7af6eb7b5464c451fbf54465ddd940f7f5a7"} err="failed to get container status \"a3a126bb5c586b3d294a5d3a5d3a7af6eb7b5464c451fbf54465ddd940f7f5a7\": rpc error: code = NotFound desc = could not find container \"a3a126bb5c586b3d294a5d3a5d3a7af6eb7b5464c451fbf54465ddd940f7f5a7\": container with ID starting with a3a126bb5c586b3d294a5d3a5d3a7af6eb7b5464c451fbf54465ddd940f7f5a7 not found: ID does not exist"
Nov 02 09:42:34.281830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:34.281805    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"]
Nov 02 09:42:34.287677 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:34.287657    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lkqpk"]
Nov 02 09:42:35.314450 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:42:35.314416    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1a4483-017b-4cb2-84e4-910f30499440" path="/var/lib/kubelet/pods/6a1a4483-017b-4cb2-84e4-910f30499440/volumes"
Nov 02 09:42:39.635447 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:42:39.635413    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:43:09.635426 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:43:09.635383    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:43:39.634959 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:43:39.634910    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:43:51.645239 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.645204    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"]
Nov 02 09:43:51.645790 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.645760    4614 cpu_manager.go:414] "containerMap: removing container" podUID="eba26412-4424-49e6-9cf2-0117a0c1ff80" containerName="maas-api"
Nov 02 09:43:51.645790 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.645781    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba26412-4424-49e6-9cf2-0117a0c1ff80" containerName="maas-api"
Nov 02 09:43:51.645871 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.645827    4614 cpu_manager.go:414] "containerMap: removing container" podUID="6a1a4483-017b-4cb2-84e4-910f30499440" containerName="manager"
Nov 02 09:43:51.645871 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.645836    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1a4483-017b-4cb2-84e4-910f30499440" containerName="manager"
Nov 02 09:43:51.645938 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.645924    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1a4483-017b-4cb2-84e4-910f30499440" containerName="manager"
Nov 02 09:43:51.645970 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.645942    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba26412-4424-49e6-9cf2-0117a0c1ff80" containerName="maas-api"
Nov 02 09:43:51.649086 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.649066    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.664435 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.664405    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"]
Nov 02 09:43:51.736012 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.735985    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b2fjp\""
Nov 02 09:43:51.740320 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.740299    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.740435 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.740331    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.740435 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.740415    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.740510 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.740438    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.740510 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.740456    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57e7275-fe40-4a86-b6e9-0939c91526b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.740570 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.740551    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.740616 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.740580    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.740657 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.740619    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k5vn\" (UniqueName: \"kubernetes.io/projected/d57e7275-fe40-4a86-b6e9-0939c91526b0-kube-api-access-4k5vn\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.740657 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.740643    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.841126 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.841094    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.841303 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.841129    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.841303 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.841161    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k5vn\" (UniqueName: \"kubernetes.io/projected/d57e7275-fe40-4a86-b6e9-0939c91526b0-kube-api-access-4k5vn\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.841303 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.841196    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.841303 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.841250    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.841303 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.841274    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.841627 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.841308    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.841627 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.841331    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.841627 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.841378    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57e7275-fe40-4a86-b6e9-0939c91526b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.841754 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.841619    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.841754 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.841737    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.841881 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.841848    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.841985 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.841960    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.842051 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.842040    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57e7275-fe40-4a86-b6e9-0939c91526b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.843681 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.843654    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.843786 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.843769    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.853472 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.853449    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.853557 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.853504    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k5vn\" (UniqueName: \"kubernetes.io/projected/d57e7275-fe40-4a86-b6e9-0939c91526b0-kube-api-access-4k5vn\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sn2pt\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:51.965854 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:51.965776    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:52.262991 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:52.262965    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"]
Nov 02 09:43:52.265848 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:43:52.265822    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57e7275_fe40_4a86_b6e9_0939c91526b0.slice/crio-dab21dda3e40eceaec2f9ce9338ec72f108f763b0482618cfc61153ce79c1569 WatchSource:0}: Error finding container dab21dda3e40eceaec2f9ce9338ec72f108f763b0482618cfc61153ce79c1569: Status 404 returned error can't find the container with id dab21dda3e40eceaec2f9ce9338ec72f108f763b0482618cfc61153ce79c1569
Nov 02 09:43:52.601028 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:52.600940    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" event={"ID":"d57e7275-fe40-4a86-b6e9-0939c91526b0","Type":"ContainerStarted","Data":"dab21dda3e40eceaec2f9ce9338ec72f108f763b0482618cfc61153ce79c1569"}
Nov 02 09:43:56.036692 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:56.036658    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:43:56.036969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:56.036728    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:43:56.036969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:56.036758    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:43:56.652130 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:56.652092    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" event={"ID":"d57e7275-fe40-4a86-b6e9-0939c91526b0","Type":"ContainerStarted","Data":"5540f746aa4e4da7f6f99005d4113e79a76ebe9baed54fd957bad95c37b42e50"}
Nov 02 09:43:56.679404 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:56.679333    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podStartSLOduration=1.9114462300000001 podStartE2EDuration="5.679321256s" totalImagesPullingTime="3.767875026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:43:56.678725098 +0000 UTC m=+517.511383177" watchObservedRunningTime="2026-11-02 09:43:56.679321256 +0000 UTC m=+517.511979336"
Nov 02 09:43:56.966503 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:56.966422    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:43:57.967731 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:57.967698    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 09:43:57.968201 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:57.967749    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 09:43:58.966526 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:58.966488    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 09:43:58.966732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:58.966545    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 09:43:59.966744 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:59.966707    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 09:43:59.967156 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:43:59.966762    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 09:44:00.966369 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:00.966311    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 09:44:00.966576 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:00.966396    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 09:44:01.967081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:01.967047    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 09:44:01.967514 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:01.967099    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 09:44:02.967123 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:02.967086    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 09:44:02.967555 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:02.967142    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 09:44:03.966649 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:03.966616    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Nov 02 09:44:03.966819 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:03.966668    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Nov 02 09:44:05.144850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:05.144802    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:44:05.145329 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:05.145218    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:44:05.146027 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:05.146007    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 09:44:06.683680 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.683644    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"]
Nov 02 09:44:06.687330 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.687307    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.716927 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.716895    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"]
Nov 02 09:44:06.804578 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.804548    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.804757 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.804588    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.804757 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.804619    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.804836 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.804793    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.804886 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.804830    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.804967 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.804942    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.805079 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.805050    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.805144 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.805100    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.805144 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.805126    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzzjg\" (UniqueName: \"kubernetes.io/projected/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-kube-api-access-zzzjg\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.905920 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.905885    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.905920 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.905930    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.906126 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.905947    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.906126 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.906064    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.906126 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.906106    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.906257 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.906139    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.906257 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.906168    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzzjg\" (UniqueName: \"kubernetes.io/projected/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-kube-api-access-zzzjg\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.906257 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.906242    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.906433 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.906276    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.906433 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.906336    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.906433 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.906423    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.906664 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.906630    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.906898 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.906874    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.907000 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.906982    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.908211 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.908193    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.908543 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.908523    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.919282 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.919254    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:06.919390 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:06.919322    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzzjg\" (UniqueName: \"kubernetes.io/projected/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-kube-api-access-zzzjg\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-vmkcq\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:07.000976 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:07.000945    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:07.306382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:07.306290    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"]
Nov 02 09:44:07.310700 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:44:07.310669    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f527771_b4d1_43ee_89b4_264cfdf7f4d0.slice/crio-94cc95c7d5ac9509ee53e67d63f596657195e29367123c7a53b5c937bc1b72d1 WatchSource:0}: Error finding container 94cc95c7d5ac9509ee53e67d63f596657195e29367123c7a53b5c937bc1b72d1: Status 404 returned error can't find the container with id 94cc95c7d5ac9509ee53e67d63f596657195e29367123c7a53b5c937bc1b72d1
Nov 02 09:44:07.314212 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:07.314174    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:44:07.314292 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:07.314261    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:44:07.314330 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:07.314308    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:44:07.707070 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:07.706977    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" event={"ID":"6f527771-b4d1-43ee-89b4-264cfdf7f4d0","Type":"ContainerStarted","Data":"2f087b93cea8c27547aaa5a52530d4a14c04daf23ed726ab7b4a8400b42a1d3e"}
Nov 02 09:44:07.707070 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:07.707063    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" event={"ID":"6f527771-b4d1-43ee-89b4-264cfdf7f4d0","Type":"ContainerStarted","Data":"94cc95c7d5ac9509ee53e67d63f596657195e29367123c7a53b5c937bc1b72d1"}
Nov 02 09:44:07.736518 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:07.736458    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" podStartSLOduration=1.736441028 podStartE2EDuration="1.736441028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:07.734948799 +0000 UTC m=+528.567606880" watchObservedRunningTime="2026-11-02 09:44:07.736441028 +0000 UTC m=+528.569099109"
Nov 02 09:44:08.001144 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:08.001098    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:08.002777 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:08.002690    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 02 09:44:08.002934 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:08.002802    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" podUID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 02 09:44:09.002162 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:09.002119    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 02 09:44:09.002665 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:09.002182    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" podUID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 02 09:44:09.658317 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:44:09.658277    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:44:10.002188 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:10.002156    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 02 09:44:10.002572 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:10.002211    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" podUID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 02 09:44:11.001976 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:11.001944    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 02 09:44:11.002147 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:11.001998    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" podUID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 02 09:44:12.002065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:12.001746    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 02 09:44:12.002065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:12.001819    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" podUID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 02 09:44:13.001261 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:13.001228    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 02 09:44:13.001455 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:13.001274    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" podUID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 02 09:44:14.001654 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:14.001620    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 02 09:44:14.002024 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:14.001676    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" podUID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 02 09:44:14.449638 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:14.449553    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"]
Nov 02 09:44:14.456382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:14.456341    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"
Nov 02 09:44:14.471091 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:14.471070    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"]
Nov 02 09:44:14.497786 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:14.497748    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtnlq\" (UniqueName: \"kubernetes.io/projected/0293c383-90a8-40bd-b429-185b81da0831-kube-api-access-qtnlq\") pod \"payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp\" (UID: \"0293c383-90a8-40bd-b429-185b81da0831\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"
Nov 02 09:44:14.497969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:14.497821    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0293c383-90a8-40bd-b429-185b81da0831-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp\" (UID: \"0293c383-90a8-40bd-b429-185b81da0831\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"
Nov 02 09:44:14.530772 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:14.530747    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-000ffb\""
Nov 02 09:44:14.530943 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:14.530838    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-000ffb-dockercfg-6fwjc\""
Nov 02 09:44:14.599860 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:14.599805    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtnlq\" (UniqueName: \"kubernetes.io/projected/0293c383-90a8-40bd-b429-185b81da0831-kube-api-access-qtnlq\") pod \"payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp\" (UID: \"0293c383-90a8-40bd-b429-185b81da0831\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"
Nov 02 09:44:14.600067 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:14.599892    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0293c383-90a8-40bd-b429-185b81da0831-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp\" (UID: \"0293c383-90a8-40bd-b429-185b81da0831\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"
Nov 02 09:44:14.601222 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:14.600905    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0293c383-90a8-40bd-b429-185b81da0831-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp\" (UID: \"0293c383-90a8-40bd-b429-185b81da0831\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"
Nov 02 09:44:14.617810 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:14.617775    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtnlq\" (UniqueName: \"kubernetes.io/projected/0293c383-90a8-40bd-b429-185b81da0831-kube-api-access-qtnlq\") pod \"payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp\" (UID: \"0293c383-90a8-40bd-b429-185b81da0831\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"
Nov 02 09:44:14.771250 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:14.771219    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"
Nov 02 09:44:15.001878 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:15.001840    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Nov 02 09:44:15.002382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:15.001917    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" podUID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Nov 02 09:44:15.063105 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:15.063079    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"]
Nov 02 09:44:15.065726 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:44:15.065694    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0293c383_90a8_40bd_b429_185b81da0831.slice/crio-05d875fbec6a0dd44e93e31ec319ee23ca2916a5ecc961e6306b8e3eff3147ff WatchSource:0}: Error finding container 05d875fbec6a0dd44e93e31ec319ee23ca2916a5ecc961e6306b8e3eff3147ff: Status 404 returned error can't find the container with id 05d875fbec6a0dd44e93e31ec319ee23ca2916a5ecc961e6306b8e3eff3147ff
Nov 02 09:44:15.771805 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:15.771724    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp" event={"ID":"0293c383-90a8-40bd-b429-185b81da0831","Type":"ContainerStarted","Data":"05d875fbec6a0dd44e93e31ec319ee23ca2916a5ecc961e6306b8e3eff3147ff"}
Nov 02 09:44:16.980557 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:16.980529    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:16.981045 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:16.980948    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:16.981934 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:16.981883    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 09:44:17.785332 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:17.785301    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp" event={"ID":"0293c383-90a8-40bd-b429-185b81da0831","Type":"ContainerStarted","Data":"8f5d91e90b4da6ffec30e89dd8032e97d39dd1a9be0646a4cad99d50eefa708f"}
Nov 02 09:44:17.785468 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:17.785402    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"
Nov 02 09:44:17.806447 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:17.806404    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp" podStartSLOduration=1.17353283 podStartE2EDuration="3.806389833s" totalImagesPullingTime="2.632857003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:17.80515932 +0000 UTC m=+538.637817396" watchObservedRunningTime="2026-11-02 09:44:17.806389833 +0000 UTC m=+538.639047913"
Nov 02 09:44:21.691028 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.690990    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"]
Nov 02 09:44:21.695772 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.695751    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.708725 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.708702    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"]
Nov 02 09:44:21.780417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.779455    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3461d99e-f048-49b4-bfa3-6557e546f58b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.780417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.779522    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.780417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.780249    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.780417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.780330    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhhbt\" (UniqueName: \"kubernetes.io/projected/3461d99e-f048-49b4-bfa3-6557e546f58b-kube-api-access-qhhbt\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.780417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.780401    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.780417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.780428    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.780893 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.780458    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.780893 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.780492    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.782567 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.781165    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.882399 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.882330    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.882603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.882399    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.882603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.882452    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhhbt\" (UniqueName: \"kubernetes.io/projected/3461d99e-f048-49b4-bfa3-6557e546f58b-kube-api-access-qhhbt\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.882603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.882485    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.882603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.882510    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.882603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.882537    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.882603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.882570    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.882943 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.882651    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.882943 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.882713    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3461d99e-f048-49b4-bfa3-6557e546f58b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.882943 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.882882    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.883279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.883233    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.883530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.883472    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.883530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.883485    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.883530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.883519    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3461d99e-f048-49b4-bfa3-6557e546f58b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.885038 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.885007    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.885324 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.885298    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.905096 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.905068    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhhbt\" (UniqueName: \"kubernetes.io/projected/3461d99e-f048-49b4-bfa3-6557e546f58b-kube-api-access-qhhbt\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:21.905794 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:21.905768    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4d57x\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:22.008136 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:22.008107    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:22.295542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:22.295504    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"]
Nov 02 09:44:22.298186 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:44:22.298155    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3461d99e_f048_49b4_bfa3_6557e546f58b.slice/crio-0144f7d547e811c26c745aebde64b9eae45a657c7755234d7013f4da52137e19 WatchSource:0}: Error finding container 0144f7d547e811c26c745aebde64b9eae45a657c7755234d7013f4da52137e19: Status 404 returned error can't find the container with id 0144f7d547e811c26c745aebde64b9eae45a657c7755234d7013f4da52137e19
Nov 02 09:44:22.300515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:22.300479    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:44:22.300586 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:22.300568    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:44:22.300648 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:22.300630    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:44:22.810665 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:22.810622    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" event={"ID":"3461d99e-f048-49b4-bfa3-6557e546f58b","Type":"ContainerStarted","Data":"253281eecce78ed19353acd7c12b163c07467640050a48aec07f21ba6439b75e"}
Nov 02 09:44:22.811053 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:22.810675    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" event={"ID":"3461d99e-f048-49b4-bfa3-6557e546f58b","Type":"ContainerStarted","Data":"0144f7d547e811c26c745aebde64b9eae45a657c7755234d7013f4da52137e19"}
Nov 02 09:44:22.843377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:22.837242    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" podStartSLOduration=1.837222529 podStartE2EDuration="1.837222529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:22.834875268 +0000 UTC m=+543.667533348" watchObservedRunningTime="2026-11-02 09:44:22.837222529 +0000 UTC m=+543.669880611"
Nov 02 09:44:23.009264 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:23.009235    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:23.010310 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:23.010285    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4d57x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 02 09:44:23.010453 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:23.010331    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" podUID="3461d99e-f048-49b4-bfa3-6557e546f58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 02 09:44:24.008824 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:24.008791    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4d57x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 02 09:44:24.009224 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:24.008843    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" podUID="3461d99e-f048-49b4-bfa3-6557e546f58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 02 09:44:25.016165 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:25.014856    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4d57x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 02 09:44:25.016165 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:25.014930    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" podUID="3461d99e-f048-49b4-bfa3-6557e546f58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 02 09:44:26.008613 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:26.008575    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4d57x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 02 09:44:26.008789 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:26.008643    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" podUID="3461d99e-f048-49b4-bfa3-6557e546f58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 02 09:44:27.008900 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.008854    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4d57x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 02 09:44:27.009343 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.008946    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" podUID="3461d99e-f048-49b4-bfa3-6557e546f58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 02 09:44:27.412878 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.412799    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"]
Nov 02 09:44:27.417428 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.417408    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.422227 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.422201    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"kube-root-ca.crt\""
Nov 02 09:44:27.444373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.442545    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"default-dockercfg-xf5qm\""
Nov 02 09:44:27.445794 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.445764    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"simulator-w3-a110ef-kserve-self-signed-certs\""
Nov 02 09:44:27.459374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.447692    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"]
Nov 02 09:44:27.522190 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.522164    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"openshift-service-ca.crt\""
Nov 02 09:44:27.558020 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.557984    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-home\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.558218 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.558052    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-tmp-dir\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.558218 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.558120    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-dshm\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.558218 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.558148    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-model-cache\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.558218 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.558187    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt9dm\" (UniqueName: \"kubernetes.io/projected/e0140d2d-0778-462c-af17-bc634966b6be-kube-api-access-zt9dm\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.558465 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.558240    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0140d2d-0778-462c-af17-bc634966b6be-tls-certs\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.658889 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.658850    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0140d2d-0778-462c-af17-bc634966b6be-tls-certs\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.659033 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.658916    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-home\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.659033 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.658962    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-tmp-dir\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.659122 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.659030    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-dshm\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.659122 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.659054    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-model-cache\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.659122 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.659091    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt9dm\" (UniqueName: \"kubernetes.io/projected/e0140d2d-0778-462c-af17-bc634966b6be-kube-api-access-zt9dm\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.659749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.659728    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-model-cache\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.659883 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.659815    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-tmp-dir\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.659935 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.659887    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-home\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.663825 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.661629    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-dshm\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.664850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.664819    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0140d2d-0778-462c-af17-bc634966b6be-tls-certs\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.674038 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.674004    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt9dm\" (UniqueName: \"kubernetes.io/projected/e0140d2d-0778-462c-af17-bc634966b6be-kube-api-access-zt9dm\") pod \"simulator-w3-a110ef-kserve-5874d9cf94-bn2f9\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") " pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:27.764032 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:27.763991    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:28.008777 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:28.008739    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4d57x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 02 09:44:28.008955 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:28.008795    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" podUID="3461d99e-f048-49b4-bfa3-6557e546f58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 02 09:44:28.050159 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:28.050080    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"]
Nov 02 09:44:28.057868 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:44:28.057833    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0140d2d_0778_462c_af17_bc634966b6be.slice/crio-7204847b56817c3a5e6c8cf99a6d0b41baed477ae62d92840e8cb2d348c7147b WatchSource:0}: Error finding container 7204847b56817c3a5e6c8cf99a6d0b41baed477ae62d92840e8cb2d348c7147b: Status 404 returned error can't find the container with id 7204847b56817c3a5e6c8cf99a6d0b41baed477ae62d92840e8cb2d348c7147b
Nov 02 09:44:28.794432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:28.794400    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"
Nov 02 09:44:28.885810 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:28.885768    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" event={"ID":"e0140d2d-0778-462c-af17-bc634966b6be","Type":"ContainerStarted","Data":"7204847b56817c3a5e6c8cf99a6d0b41baed477ae62d92840e8cb2d348c7147b"}
Nov 02 09:44:29.009332 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:29.009285    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4d57x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 02 09:44:29.009858 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:29.009388    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" podUID="3461d99e-f048-49b4-bfa3-6557e546f58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 02 09:44:29.895693 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:29.895595    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" event={"ID":"e0140d2d-0778-462c-af17-bc634966b6be","Type":"ContainerStarted","Data":"f4188f79adda623ff74dc4f6ee50741fc910b79000626b492dbfc1d01c890a63"}
Nov 02 09:44:29.917802 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:29.917742    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podStartSLOduration=1.4446467969999999 podStartE2EDuration="2.917723399s" totalImagesPullingTime="1.473076602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:29.915785499 +0000 UTC m=+550.748443584" watchObservedRunningTime="2026-11-02 09:44:29.917723399 +0000 UTC m=+550.750381484"
Nov 02 09:44:30.010034 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:30.009975    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4d57x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Nov 02 09:44:30.010532 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:30.010038    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" podUID="3461d99e-f048-49b4-bfa3-6557e546f58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Nov 02 09:44:30.764227 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:30.764138    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:32.053489 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:32.053447    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4d57x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:32.054084 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:32.054051    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" podUID="3461d99e-f048-49b4-bfa3-6557e546f58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:32.396428 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:32.394719    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:32.396428 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:32.396042    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:32.398395 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:32.397161    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:44:33.029382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:33.025486    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:44:33.029382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:33.025547    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" podUID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:44:34.740846 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.739190    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"]
Nov 02 09:44:34.743338 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.743310    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.763463 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.763432    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2e80fb\"/\"kube-root-ca.crt\""
Nov 02 09:44:34.764142 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.763811    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2e80fb\"/\"default-dockercfg-gkpxc\""
Nov 02 09:44:34.764287 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.763869    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2e80fb\"/\"openshift-service-ca.crt\""
Nov 02 09:44:34.764406 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.764071    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2e80fb\"/\"simulator-w0-2e80fb-kserve-self-signed-certs\""
Nov 02 09:44:34.766743 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.766720    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"]
Nov 02 09:44:34.887728 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.887695    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-model-cache\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.887728 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.887729    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-tmp-dir\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.887728 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.887745    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-home\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.888049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.887769    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4x8\" (UniqueName: \"kubernetes.io/projected/86af445e-d01b-4537-b31a-1a78b30099c9-kube-api-access-vn4x8\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.888049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.887787    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-dshm\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.888177 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.888102    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86af445e-d01b-4537-b31a-1a78b30099c9-tls-certs\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.990085 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.989604    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86af445e-d01b-4537-b31a-1a78b30099c9-tls-certs\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.990085 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.989666    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-model-cache\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.990085 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.989688    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-tmp-dir\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.990085 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.989713    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-home\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.990085 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.989749    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4x8\" (UniqueName: \"kubernetes.io/projected/86af445e-d01b-4537-b31a-1a78b30099c9-kube-api-access-vn4x8\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.990085 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.989779    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-dshm\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.990578 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.990450    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-model-cache\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.990578 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.990483    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-tmp-dir\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.994417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.990832    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-home\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.994417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.993250    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86af445e-d01b-4537-b31a-1a78b30099c9-tls-certs\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:34.994417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:34.993528    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-dshm\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:35.005406 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:35.005384    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4x8\" (UniqueName: \"kubernetes.io/projected/86af445e-d01b-4537-b31a-1a78b30099c9-kube-api-access-vn4x8\") pod \"simulator-w0-2e80fb-kserve-555977f8d-dh8rz\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:35.060764 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:35.059545    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:35.396400 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:44:35.396323    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86af445e_d01b_4537_b31a_1a78b30099c9.slice/crio-94384f9748d31e98043b06dac2b97f6952b0f5d488164a6262e84c40a8c58349 WatchSource:0}: Error finding container 94384f9748d31e98043b06dac2b97f6952b0f5d488164a6262e84c40a8c58349: Status 404 returned error can't find the container with id 94384f9748d31e98043b06dac2b97f6952b0f5d488164a6262e84c40a8c58349
Nov 02 09:44:35.398185 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:35.398160    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"]
Nov 02 09:44:35.937553 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:35.937465    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" event={"ID":"86af445e-d01b-4537-b31a-1a78b30099c9","Type":"ContainerStarted","Data":"e6eb4054aea731ca7e7f0ad89b67e55ff146bfb5ee06c56f4ada47cc07ced885"}
Nov 02 09:44:35.937553 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:35.937521    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" event={"ID":"86af445e-d01b-4537-b31a-1a78b30099c9","Type":"ContainerStarted","Data":"94384f9748d31e98043b06dac2b97f6952b0f5d488164a6262e84c40a8c58349"}
Nov 02 09:44:35.960135 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:35.959677    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podStartSLOduration=1.95965771 podStartE2EDuration="1.95965771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:35.957331293 +0000 UTC m=+556.789989370" watchObservedRunningTime="2026-11-02 09:44:35.95965771 +0000 UTC m=+556.792315792"
Nov 02 09:44:36.060296 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:36.060257    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:37.771883 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:37.770413    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:37.812141 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:37.811701    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:38.020729 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:38.020688    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:44:39.648888 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:44:39.648761    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:44:44.879291 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:44.879251    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"]
Nov 02 09:44:44.883226 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:44.883205    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:44.894659 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:44.894633    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"premium-w3-a110ef-kserve-self-signed-certs\""
Nov 02 09:44:44.894787 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:44.894717    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"]
Nov 02 09:44:44.965856 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:44.965817    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-model-cache\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:44.965856 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:44.965848    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-tmp-dir\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:44.966117 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:44.965879    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrrvf\" (UniqueName: \"kubernetes.io/projected/577534f3-6ae1-40fe-a27f-29084e674662-kube-api-access-jrrvf\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:44.966117 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:44.966035    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-dshm\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:44.966117 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:44.966102    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-home\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:44.966288 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:44.966151    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/577534f3-6ae1-40fe-a27f-29084e674662-tls-certs\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:45.060450 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.060419    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:45.067440 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.067414    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-home\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:45.067589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.067449    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/577534f3-6ae1-40fe-a27f-29084e674662-tls-certs\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:45.067589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.067488    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-model-cache\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:45.067589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.067511    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-tmp-dir\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:45.067589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.067555    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrrvf\" (UniqueName: \"kubernetes.io/projected/577534f3-6ae1-40fe-a27f-29084e674662-kube-api-access-jrrvf\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:45.067805 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.067623    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-dshm\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:45.067902 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.067875    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-home\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:45.068083 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.067892    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-tmp-dir\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:45.068083 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.067964    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-model-cache\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:45.069831 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.069811    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-dshm\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:45.070101 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.070082    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/577534f3-6ae1-40fe-a27f-29084e674662-tls-certs\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:45.073903 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.073880    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:45.080861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.080837    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrrvf\" (UniqueName: \"kubernetes.io/projected/577534f3-6ae1-40fe-a27f-29084e674662-kube-api-access-jrrvf\") pod \"premium-w3-a110ef-kserve-784d885ff9-ck4kl\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") " pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:45.198799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.198722    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:45.482090 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:45.482061    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"]
Nov 02 09:44:46.056990 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:46.056950    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" event={"ID":"577534f3-6ae1-40fe-a27f-29084e674662","Type":"ContainerStarted","Data":"efad52f5b3aa5bc5e8f8a3d1b6ad2620f7cc183d05d90f1624f6799644e6ad4e"}
Nov 02 09:44:46.057426 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:46.057003    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" event={"ID":"577534f3-6ae1-40fe-a27f-29084e674662","Type":"ContainerStarted","Data":"cfb142e0c1e6f408053f79d44df8317fe2e9ab04a3c4ecf0a54d30b974ee1163"}
Nov 02 09:44:46.069047 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:46.069020    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:44:46.079426 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:46.079383    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podStartSLOduration=2.079345529 podStartE2EDuration="2.079345529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:46.077709047 +0000 UTC m=+566.910367129" watchObservedRunningTime="2026-11-02 09:44:46.079345529 +0000 UTC m=+566.912003602"
Nov 02 09:44:46.199301 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:46.199271    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:54.522411 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.522375    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"]
Nov 02 09:44:54.530484 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.530458    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.539688 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.539654    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"]
Nov 02 09:44:54.541896 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.541167    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2e80fb\"/\"premium-w0-2e80fb-kserve-self-signed-certs\""
Nov 02 09:44:54.569479 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.569430    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-tmp-dir\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.569670 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.569496    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-model-cache\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.569670 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.569566    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zntwk\" (UniqueName: \"kubernetes.io/projected/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-kube-api-access-zntwk\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.569670 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.569663    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-dshm\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.569924 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.569713    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-home\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.569924 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.569746    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-tls-certs\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.673961 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.673913    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-dshm\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.674241 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.674220    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-home\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.674408 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.674385    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-tls-certs\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.674556 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.674542    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-tmp-dir\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.674670 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.674658    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-model-cache\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.674813 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.674796    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zntwk\" (UniqueName: \"kubernetes.io/projected/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-kube-api-access-zntwk\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.675608 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.675585    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-home\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.676595 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.676572    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-tmp-dir\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.677119 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.677010    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-model-cache\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.699346 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.699312    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zntwk\" (UniqueName: \"kubernetes.io/projected/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-kube-api-access-zntwk\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.707125 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.707067    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-tls-certs\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.711624 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.711530    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-dshm\") pod \"premium-w0-2e80fb-kserve-66d5b55789-tbjbd\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") " pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:54.851051 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:54.850963    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:44:55.131785 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:55.131761    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"]
Nov 02 09:44:55.134286 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:44:55.134250    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded9eb91a_cdf9_49bf_8b73_329fe059bc42.slice/crio-b896bc4aa45dd469dfe441e53eb1d92c39c63397ab0baa14a945785439436a61 WatchSource:0}: Error finding container b896bc4aa45dd469dfe441e53eb1d92c39c63397ab0baa14a945785439436a61: Status 404 returned error can't find the container with id b896bc4aa45dd469dfe441e53eb1d92c39c63397ab0baa14a945785439436a61
Nov 02 09:44:55.198988 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:55.198962    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:55.217986 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:55.217957    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:56.128048 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:56.128015    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" event={"ID":"ed9eb91a-cdf9-49bf-8b73-329fe059bc42","Type":"ContainerStarted","Data":"27b26f488ef408cd29d6998efc6b2a5dda1e845c940af829a54fb494f144dc74"}
Nov 02 09:44:56.128577 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:56.128068    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" event={"ID":"ed9eb91a-cdf9-49bf-8b73-329fe059bc42","Type":"ContainerStarted","Data":"b896bc4aa45dd469dfe441e53eb1d92c39c63397ab0baa14a945785439436a61"}
Nov 02 09:44:56.139780 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:56.139754    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:44:56.149437 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:56.149384    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podStartSLOduration=2.149367114 podStartE2EDuration="2.149367114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:44:56.147911974 +0000 UTC m=+576.980570054" watchObservedRunningTime="2026-11-02 09:44:56.149367114 +0000 UTC m=+576.982025187"
Nov 02 09:44:56.851629 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:44:56.851597    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:45:00.149212 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.149127    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"]
Nov 02 09:45:00.156066 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.156038    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"]
Nov 02 09:45:00.156226 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.156172    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"
Nov 02 09:45:00.159044 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.159024    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"
Nov 02 09:45:00.162010 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.161985    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"]
Nov 02 09:45:00.165594 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.165571    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:45:00.168484 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.168462    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"]
Nov 02 09:45:00.170671 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.170650    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"]
Nov 02 09:45:00.178928 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.178114    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"
Nov 02 09:45:00.187437 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.187391    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"]
Nov 02 09:45:00.240113 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.240080    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/415fabd5-bf72-45b4-b510-5e473469f6ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893545-jv9p4\" (UID: \"415fabd5-bf72-45b4-b510-5e473469f6ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"
Nov 02 09:45:00.240113 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.240118    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf8cl\" (UniqueName: \"kubernetes.io/projected/415fabd5-bf72-45b4-b510-5e473469f6ce-kube-api-access-zf8cl\") pod \"maas-api-key-cleanup-29893545-jv9p4\" (UID: \"415fabd5-bf72-45b4-b510-5e473469f6ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"
Nov 02 09:45:00.240341 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.240238    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh\" (UID: \"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"
Nov 02 09:45:00.240341 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.240270    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdtbr\" (UniqueName: \"kubernetes.io/projected/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd-kube-api-access-tdtbr\") pod \"maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh\" (UID: \"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"
Nov 02 09:45:00.240341 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.240291    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c23a6d-c1aa-4fb5-9e57-6faaa004af68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446\" (UID: \"30c23a6d-c1aa-4fb5-9e57-6faaa004af68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"
Nov 02 09:45:00.240341 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.240311    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52jnm\" (UniqueName: \"kubernetes.io/projected/30c23a6d-c1aa-4fb5-9e57-6faaa004af68-kube-api-access-52jnm\") pod \"maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446\" (UID: \"30c23a6d-c1aa-4fb5-9e57-6faaa004af68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"
Nov 02 09:45:00.341792 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.341756    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdtbr\" (UniqueName: \"kubernetes.io/projected/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd-kube-api-access-tdtbr\") pod \"maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh\" (UID: \"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"
Nov 02 09:45:00.341792 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.341795    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c23a6d-c1aa-4fb5-9e57-6faaa004af68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446\" (UID: \"30c23a6d-c1aa-4fb5-9e57-6faaa004af68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"
Nov 02 09:45:00.342047 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.341817    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52jnm\" (UniqueName: \"kubernetes.io/projected/30c23a6d-c1aa-4fb5-9e57-6faaa004af68-kube-api-access-52jnm\") pod \"maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446\" (UID: \"30c23a6d-c1aa-4fb5-9e57-6faaa004af68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"
Nov 02 09:45:00.342047 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.341859    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/415fabd5-bf72-45b4-b510-5e473469f6ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893545-jv9p4\" (UID: \"415fabd5-bf72-45b4-b510-5e473469f6ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"
Nov 02 09:45:00.342047 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.341968    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8cl\" (UniqueName: \"kubernetes.io/projected/415fabd5-bf72-45b4-b510-5e473469f6ce-kube-api-access-zf8cl\") pod \"maas-api-key-cleanup-29893545-jv9p4\" (UID: \"415fabd5-bf72-45b4-b510-5e473469f6ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"
Nov 02 09:45:00.342170 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.342075    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh\" (UID: \"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"
Nov 02 09:45:00.342617 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.342598    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c23a6d-c1aa-4fb5-9e57-6faaa004af68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446\" (UID: \"30c23a6d-c1aa-4fb5-9e57-6faaa004af68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"
Nov 02 09:45:00.342669 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.342602    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/415fabd5-bf72-45b4-b510-5e473469f6ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893545-jv9p4\" (UID: \"415fabd5-bf72-45b4-b510-5e473469f6ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"
Nov 02 09:45:00.342842 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.342823    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh\" (UID: \"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"
Nov 02 09:45:00.356737 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.356710    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52jnm\" (UniqueName: \"kubernetes.io/projected/30c23a6d-c1aa-4fb5-9e57-6faaa004af68-kube-api-access-52jnm\") pod \"maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446\" (UID: \"30c23a6d-c1aa-4fb5-9e57-6faaa004af68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"
Nov 02 09:45:00.356859 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.356717    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdtbr\" (UniqueName: \"kubernetes.io/projected/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd-kube-api-access-tdtbr\") pod \"maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh\" (UID: \"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"
Nov 02 09:45:00.356859 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.356829    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf8cl\" (UniqueName: \"kubernetes.io/projected/415fabd5-bf72-45b4-b510-5e473469f6ce-kube-api-access-zf8cl\") pod \"maas-api-key-cleanup-29893545-jv9p4\" (UID: \"415fabd5-bf72-45b4-b510-5e473469f6ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"
Nov 02 09:45:00.488217 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.488186    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"
Nov 02 09:45:00.493591 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.493545    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"
Nov 02 09:45:00.498654 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.498629    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"
Nov 02 09:45:00.780908 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.780879    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"]
Nov 02 09:45:00.782570 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:45:00.782541    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d745e8e_2ca3_4625_b6a6_e89bb9b49afd.slice/crio-449adadd0a939a949088eb588cb599940b5737fddc113a3fee499e21ac4b8ed3 WatchSource:0}: Error finding container 449adadd0a939a949088eb588cb599940b5737fddc113a3fee499e21ac4b8ed3: Status 404 returned error can't find the container with id 449adadd0a939a949088eb588cb599940b5737fddc113a3fee499e21ac4b8ed3
Nov 02 09:45:00.819531 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.819390    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"]
Nov 02 09:45:00.821859 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:45:00.821830    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c23a6d_c1aa_4fb5_9e57_6faaa004af68.slice/crio-4709188e7068a294294168a7bad252ab324798ebfa763c2083e2250e0e506147 WatchSource:0}: Error finding container 4709188e7068a294294168a7bad252ab324798ebfa763c2083e2250e0e506147: Status 404 returned error can't find the container with id 4709188e7068a294294168a7bad252ab324798ebfa763c2083e2250e0e506147
Nov 02 09:45:00.822297 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:00.822276    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"]
Nov 02 09:45:00.823519 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:45:00.823381    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415fabd5_bf72_45b4_b510_5e473469f6ce.slice/crio-93b07af0351715c64496e51ca3fef3f64008f1faf8bc6ce870e1cc01906a4d34 WatchSource:0}: Error finding container 93b07af0351715c64496e51ca3fef3f64008f1faf8bc6ce870e1cc01906a4d34: Status 404 returned error can't find the container with id 93b07af0351715c64496e51ca3fef3f64008f1faf8bc6ce870e1cc01906a4d34
Nov 02 09:45:01.154945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.154855    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh" event={"ID":"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd","Type":"ContainerStarted","Data":"449adadd0a939a949088eb588cb599940b5737fddc113a3fee499e21ac4b8ed3"}
Nov 02 09:45:01.156140 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.156115    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446" event={"ID":"30c23a6d-c1aa-4fb5-9e57-6faaa004af68","Type":"ContainerStarted","Data":"4709188e7068a294294168a7bad252ab324798ebfa763c2083e2250e0e506147"}
Nov 02 09:45:01.157395 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.157374    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4" event={"ID":"415fabd5-bf72-45b4-b510-5e473469f6ce","Type":"ContainerStarted","Data":"93b07af0351715c64496e51ca3fef3f64008f1faf8bc6ce870e1cc01906a4d34"}
Nov 02 09:45:01.730213 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.730181    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"]
Nov 02 09:45:01.735518 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.735495    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.745603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.745069    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-000ffb\"/\"openshift-service-ca.crt\""
Nov 02 09:45:01.745603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.745140    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-000ffb\"/\"kube-root-ca.crt\""
Nov 02 09:45:01.745603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.745080    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-000ffb\"/\"praxis-route-000ffb-kserve-self-signed-certs\""
Nov 02 09:45:01.745603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.745175    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-000ffb\"/\"default-dockercfg-k8qc7\""
Nov 02 09:45:01.747329 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.747304    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"]
Nov 02 09:45:01.856405 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.856347    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zmp2\" (UniqueName: \"kubernetes.io/projected/f0d14fe7-3f66-4328-a2ef-535f741c1c05-kube-api-access-8zmp2\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.856405 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.856408    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d14fe7-3f66-4328-a2ef-535f741c1c05-tls-certs\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.856681 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.856433    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-tmp-dir\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.856681 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.856571    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-home\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.856681 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.856642    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-dshm\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.856842 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.856715    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-model-cache\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.957560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.957522    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-home\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.957735 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.957573    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-dshm\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.957735 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.957615    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-model-cache\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.957735 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.957676    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zmp2\" (UniqueName: \"kubernetes.io/projected/f0d14fe7-3f66-4328-a2ef-535f741c1c05-kube-api-access-8zmp2\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.957735 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.957698    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d14fe7-3f66-4328-a2ef-535f741c1c05-tls-certs\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.957735 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.957716    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-tmp-dir\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.958046 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.958020    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-home\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.958120 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.958106    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-model-cache\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.958168 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.958136    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-tmp-dir\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.959943 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.959919    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-dshm\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.960265 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.960245    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d14fe7-3f66-4328-a2ef-535f741c1c05-tls-certs\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:01.970147 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:01.970122    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zmp2\" (UniqueName: \"kubernetes.io/projected/f0d14fe7-3f66-4328-a2ef-535f741c1c05-kube-api-access-8zmp2\") pod \"praxis-route-000ffb-kserve-5d889b6685-p7ltw\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") " pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:02.048519 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:02.048430    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:02.333123 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:02.333093    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"]
Nov 02 09:45:02.335708 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:45:02.335678    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0d14fe7_3f66_4328_a2ef_535f741c1c05.slice/crio-3950e36b9813eb03c8732bc5106380031378a6c557383744d37a36ddce574707 WatchSource:0}: Error finding container 3950e36b9813eb03c8732bc5106380031378a6c557383744d37a36ddce574707: Status 404 returned error can't find the container with id 3950e36b9813eb03c8732bc5106380031378a6c557383744d37a36ddce574707
Nov 02 09:45:03.171240 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:03.170773    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw" event={"ID":"f0d14fe7-3f66-4328-a2ef-535f741c1c05","Type":"ContainerStarted","Data":"0d0d74945286d9d02f505015b161f35c28de37752e637990882bd9448d78b5d2"}
Nov 02 09:45:03.171240 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:03.170835    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw" event={"ID":"f0d14fe7-3f66-4328-a2ef-535f741c1c05","Type":"ContainerStarted","Data":"3950e36b9813eb03c8732bc5106380031378a6c557383744d37a36ddce574707"}
Nov 02 09:45:03.198261 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:03.198193    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw" podStartSLOduration=2.198174553 podStartE2EDuration="2.198174553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:03.193662536 +0000 UTC m=+584.026320617" watchObservedRunningTime="2026-11-02 09:45:03.198174553 +0000 UTC m=+584.030832636"
Nov 02 09:45:04.048945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:04.048877    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:04.852001 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:04.851967    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:45:04.866595 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:04.866571    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:45:05.184489 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:05.184452    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4" event={"ID":"415fabd5-bf72-45b4-b510-5e473469f6ce","Type":"ContainerStarted","Data":"1245cf921955e165c155f84a1b337cc0c012f628456d31431c603ba3ed4661b0"}
Nov 02 09:45:05.186048 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:05.186019    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh" event={"ID":"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd","Type":"ContainerStarted","Data":"6a2914c385ac539dcbf7c7a722e0cf8125475f00ece132e5440cd1e9d02e73fd"}
Nov 02 09:45:05.197486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:05.197466    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:45:05.201922 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:05.201874    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4" podStartSLOduration=1.6040567449999998 podStartE2EDuration="5.201851427s" totalImagesPullingTime="3.597794682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:05.200825094 +0000 UTC m=+586.033483188" watchObservedRunningTime="2026-11-02 09:45:05.201851427 +0000 UTC m=+586.034509556"
Nov 02 09:45:06.195519 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:06.195419    4614 generic.go:356] "Generic (PLEG): container finished" podID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" containerID="6a2914c385ac539dcbf7c7a722e0cf8125475f00ece132e5440cd1e9d02e73fd" exitCode=60
Nov 02 09:45:06.195979 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:06.195510    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh" event={"ID":"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd","Type":"ContainerDied","Data":"6a2914c385ac539dcbf7c7a722e0cf8125475f00ece132e5440cd1e9d02e73fd"}
Nov 02 09:45:06.195979 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:06.195729    4614 scope.go:122] "RemoveContainer" containerID="6a2914c385ac539dcbf7c7a722e0cf8125475f00ece132e5440cd1e9d02e73fd"
Nov 02 09:45:06.197280 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:06.197253    4614 generic.go:356] "Generic (PLEG): container finished" podID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68" containerID="0db2d84fff1c91a037ff7e0da66ae687999f1f58fe636ae250ab1bc328244675" exitCode=60
Nov 02 09:45:06.197390 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:06.197284    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446" event={"ID":"30c23a6d-c1aa-4fb5-9e57-6faaa004af68","Type":"ContainerDied","Data":"0db2d84fff1c91a037ff7e0da66ae687999f1f58fe636ae250ab1bc328244675"}
Nov 02 09:45:06.197719 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:06.197701    4614 scope.go:122] "RemoveContainer" containerID="0db2d84fff1c91a037ff7e0da66ae687999f1f58fe636ae250ab1bc328244675"
Nov 02 09:45:06.199199 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:06.199161    4614 generic.go:356] "Generic (PLEG): container finished" podID="415fabd5-bf72-45b4-b510-5e473469f6ce" containerID="1245cf921955e165c155f84a1b337cc0c012f628456d31431c603ba3ed4661b0" exitCode=60
Nov 02 09:45:06.199297 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:06.199217    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4" event={"ID":"415fabd5-bf72-45b4-b510-5e473469f6ce","Type":"ContainerDied","Data":"1245cf921955e165c155f84a1b337cc0c012f628456d31431c603ba3ed4661b0"}
Nov 02 09:45:06.199470 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:06.199452    4614 scope.go:122] "RemoveContainer" containerID="1245cf921955e165c155f84a1b337cc0c012f628456d31431c603ba3ed4661b0"
Nov 02 09:45:07.206603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:07.206568    4614 generic.go:356] "Generic (PLEG): container finished" podID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68" containerID="fca93ee0647b3a29e8671a1ef400b19de52ab4d40f5903a35169d3217c55fb3e" exitCode=60
Nov 02 09:45:07.207056 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:07.206661    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446" event={"ID":"30c23a6d-c1aa-4fb5-9e57-6faaa004af68","Type":"ContainerDied","Data":"fca93ee0647b3a29e8671a1ef400b19de52ab4d40f5903a35169d3217c55fb3e"}
Nov 02 09:45:07.207056 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:07.206733    4614 scope.go:122] "RemoveContainer" containerID="0db2d84fff1c91a037ff7e0da66ae687999f1f58fe636ae250ab1bc328244675"
Nov 02 09:45:07.207056 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:07.206967    4614 scope.go:122] "RemoveContainer" containerID="fca93ee0647b3a29e8671a1ef400b19de52ab4d40f5903a35169d3217c55fb3e"
Nov 02 09:45:07.207255 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:45:07.207233    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446_odh-ai-gateway-infra(30c23a6d-c1aa-4fb5-9e57-6faaa004af68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446" podUID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68"
Nov 02 09:45:07.208666 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:07.208644    4614 generic.go:356] "Generic (PLEG): container finished" podID="415fabd5-bf72-45b4-b510-5e473469f6ce" containerID="728c8dbd7cbeaf285a101f95bbd843acd60db9c717a092dc8ca97a01e49a158a" exitCode=60
Nov 02 09:45:07.208762 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:07.208726    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4" event={"ID":"415fabd5-bf72-45b4-b510-5e473469f6ce","Type":"ContainerDied","Data":"728c8dbd7cbeaf285a101f95bbd843acd60db9c717a092dc8ca97a01e49a158a"}
Nov 02 09:45:07.208957 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:07.208941    4614 scope.go:122] "RemoveContainer" containerID="728c8dbd7cbeaf285a101f95bbd843acd60db9c717a092dc8ca97a01e49a158a"
Nov 02 09:45:07.209169 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:45:07.209146    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893545-jv9p4_odh-ai-gateway-infra(415fabd5-bf72-45b4-b510-5e473469f6ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4" podUID="415fabd5-bf72-45b4-b510-5e473469f6ce"
Nov 02 09:45:07.210524 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:07.210497    4614 generic.go:356] "Generic (PLEG): container finished" podID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" containerID="5af1bd80ad0b923e8755f1ad1ba2864763f1541e3b9b0c0aeb66dee9c71fff65" exitCode=60
Nov 02 09:45:07.210524 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:07.210523    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh" event={"ID":"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd","Type":"ContainerDied","Data":"5af1bd80ad0b923e8755f1ad1ba2864763f1541e3b9b0c0aeb66dee9c71fff65"}
Nov 02 09:45:07.210728 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:07.210714    4614 scope.go:122] "RemoveContainer" containerID="5af1bd80ad0b923e8755f1ad1ba2864763f1541e3b9b0c0aeb66dee9c71fff65"
Nov 02 09:45:07.210912 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:45:07.210895    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh_odh-ai-gateway-infra(6d745e8e-2ca3-4625-b6a6-e89bb9b49afd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh" podUID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd"
Nov 02 09:45:07.218576 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:07.218560    4614 scope.go:122] "RemoveContainer" containerID="1245cf921955e165c155f84a1b337cc0c012f628456d31431c603ba3ed4661b0"
Nov 02 09:45:07.230303 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:07.230284    4614 scope.go:122] "RemoveContainer" containerID="6a2914c385ac539dcbf7c7a722e0cf8125475f00ece132e5440cd1e9d02e73fd"
Nov 02 09:45:08.218208 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:08.218177    4614 scope.go:122] "RemoveContainer" containerID="728c8dbd7cbeaf285a101f95bbd843acd60db9c717a092dc8ca97a01e49a158a"
Nov 02 09:45:08.218695 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:45:08.218465    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893545-jv9p4_odh-ai-gateway-infra(415fabd5-bf72-45b4-b510-5e473469f6ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4" podUID="415fabd5-bf72-45b4-b510-5e473469f6ce"
Nov 02 09:45:08.220011 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:08.219978    4614 scope.go:122] "RemoveContainer" containerID="5af1bd80ad0b923e8755f1ad1ba2864763f1541e3b9b0c0aeb66dee9c71fff65"
Nov 02 09:45:08.220219 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:45:08.220193    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh_odh-ai-gateway-infra(6d745e8e-2ca3-4625-b6a6-e89bb9b49afd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh" podUID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd"
Nov 02 09:45:08.221762 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:08.221739    4614 scope.go:122] "RemoveContainer" containerID="fca93ee0647b3a29e8671a1ef400b19de52ab4d40f5903a35169d3217c55fb3e"
Nov 02 09:45:08.222010 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:45:08.221986    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446_odh-ai-gateway-infra(30c23a6d-c1aa-4fb5-9e57-6faaa004af68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446" podUID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68"
Nov 02 09:45:09.637159 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:45:09.637119    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:45:12.048773 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:12.048735    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:12.061852 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:12.061824    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:12.258912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:12.258882    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:45:17.314898 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:17.314861    4614 scope.go:122] "RemoveContainer" containerID="728c8dbd7cbeaf285a101f95bbd843acd60db9c717a092dc8ca97a01e49a158a"
Nov 02 09:45:18.296771 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:18.296736    4614 generic.go:356] "Generic (PLEG): container finished" podID="415fabd5-bf72-45b4-b510-5e473469f6ce" containerID="6d8a9f694ae5e051fc7a896d2e0e15c09ff32379aaa426137acd2baaf0877c56" exitCode=60
Nov 02 09:45:18.296907 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:18.296805    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4" event={"ID":"415fabd5-bf72-45b4-b510-5e473469f6ce","Type":"ContainerDied","Data":"6d8a9f694ae5e051fc7a896d2e0e15c09ff32379aaa426137acd2baaf0877c56"}
Nov 02 09:45:18.296907 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:18.296892    4614 scope.go:122] "RemoveContainer" containerID="728c8dbd7cbeaf285a101f95bbd843acd60db9c717a092dc8ca97a01e49a158a"
Nov 02 09:45:18.297199 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:18.297180    4614 scope.go:122] "RemoveContainer" containerID="6d8a9f694ae5e051fc7a896d2e0e15c09ff32379aaa426137acd2baaf0877c56"
Nov 02 09:45:18.297484 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:45:18.297460    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893545-jv9p4_odh-ai-gateway-infra(415fabd5-bf72-45b4-b510-5e473469f6ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4" podUID="415fabd5-bf72-45b4-b510-5e473469f6ce"
Nov 02 09:45:19.331004 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:19.330972    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"]
Nov 02 09:45:19.369516 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:19.369480    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 09:45:19.370530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:19.370497    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 09:45:19.486043 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:19.486019    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"
Nov 02 09:45:19.544935 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:19.544905    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415fabd5-bf72-45b4-b510-5e473469f6ce-kube-api-access-zf8cl\" (UniqueName: \"kubernetes.io/projected/415fabd5-bf72-45b4-b510-5e473469f6ce-kube-api-access-zf8cl\") pod \"415fabd5-bf72-45b4-b510-5e473469f6ce\" (UID: \"415fabd5-bf72-45b4-b510-5e473469f6ce\") "
Nov 02 09:45:19.545094 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:19.544960    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/415fabd5-bf72-45b4-b510-5e473469f6ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/415fabd5-bf72-45b4-b510-5e473469f6ce-maas-api-service-ca\") pod \"415fabd5-bf72-45b4-b510-5e473469f6ce\" (UID: \"415fabd5-bf72-45b4-b510-5e473469f6ce\") "
Nov 02 09:45:19.545328 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:19.545305    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415fabd5-bf72-45b4-b510-5e473469f6ce-maas-api-service-ca" pod "415fabd5-bf72-45b4-b510-5e473469f6ce" (UID: "415fabd5-bf72-45b4-b510-5e473469f6ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:45:19.547023 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:19.547002    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415fabd5-bf72-45b4-b510-5e473469f6ce-kube-api-access-zf8cl" pod "415fabd5-bf72-45b4-b510-5e473469f6ce" (UID: "415fabd5-bf72-45b4-b510-5e473469f6ce"). InnerVolumeSpecName "kube-api-access-zf8cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:45:19.645988 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:19.645905    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/415fabd5-bf72-45b4-b510-5e473469f6ce-maas-api-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:19.645988 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:19.645935    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf8cl\" (UniqueName: \"kubernetes.io/projected/415fabd5-bf72-45b4-b510-5e473469f6ce-kube-api-access-zf8cl\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:20.308590 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:20.308561    4614 scope.go:122] "RemoveContainer" containerID="fca93ee0647b3a29e8671a1ef400b19de52ab4d40f5903a35169d3217c55fb3e"
Nov 02 09:45:20.311495 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:20.311457    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"
Nov 02 09:45:20.311560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:20.311462    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4" event={"ID":"415fabd5-bf72-45b4-b510-5e473469f6ce","Type":"ContainerDied","Data":"93b07af0351715c64496e51ca3fef3f64008f1faf8bc6ce870e1cc01906a4d34"}
Nov 02 09:45:20.311599 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:20.311590    4614 scope.go:122] "RemoveContainer" containerID="6d8a9f694ae5e051fc7a896d2e0e15c09ff32379aaa426137acd2baaf0877c56"
Nov 02 09:45:20.351183 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:20.351152    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"]
Nov 02 09:45:20.368788 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:20.368764    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893545-jv9p4"]
Nov 02 09:45:21.314882 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:21.314850    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="415fabd5-bf72-45b4-b510-5e473469f6ce" path="/var/lib/kubelet/pods/415fabd5-bf72-45b4-b510-5e473469f6ce/volumes"
Nov 02 09:45:21.315588 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:21.315567    4614 scope.go:122] "RemoveContainer" containerID="5af1bd80ad0b923e8755f1ad1ba2864763f1541e3b9b0c0aeb66dee9c71fff65"
Nov 02 09:45:21.318419 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:21.318371    4614 generic.go:356] "Generic (PLEG): container finished" podID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68" containerID="bc5b9e3609f2dd86b182eaafcf4964781d3b62c4d156de58c31f9b548cf63ffe" exitCode=60
Nov 02 09:45:21.318486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:21.318443    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446" event={"ID":"30c23a6d-c1aa-4fb5-9e57-6faaa004af68","Type":"ContainerDied","Data":"bc5b9e3609f2dd86b182eaafcf4964781d3b62c4d156de58c31f9b548cf63ffe"}
Nov 02 09:45:21.318624 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:21.318514    4614 scope.go:122] "RemoveContainer" containerID="fca93ee0647b3a29e8671a1ef400b19de52ab4d40f5903a35169d3217c55fb3e"
Nov 02 09:45:21.318798 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:21.318780    4614 scope.go:122] "RemoveContainer" containerID="bc5b9e3609f2dd86b182eaafcf4964781d3b62c4d156de58c31f9b548cf63ffe"
Nov 02 09:45:21.319132 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:45:21.319109    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446_odh-ai-gateway-infra(30c23a6d-c1aa-4fb5-9e57-6faaa004af68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446" podUID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68"
Nov 02 09:45:22.327222 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:22.327178    4614 generic.go:356] "Generic (PLEG): container finished" podID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" containerID="5c4f68c17f8c722ef1edcae5afc841286f14ae6fe466706e91e017f4c24e4ac7" exitCode=60
Nov 02 09:45:22.327733 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:22.327254    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh" event={"ID":"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd","Type":"ContainerDied","Data":"5c4f68c17f8c722ef1edcae5afc841286f14ae6fe466706e91e017f4c24e4ac7"}
Nov 02 09:45:22.327733 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:22.327333    4614 scope.go:122] "RemoveContainer" containerID="5af1bd80ad0b923e8755f1ad1ba2864763f1541e3b9b0c0aeb66dee9c71fff65"
Nov 02 09:45:22.327733 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:22.327714    4614 scope.go:122] "RemoveContainer" containerID="5c4f68c17f8c722ef1edcae5afc841286f14ae6fe466706e91e017f4c24e4ac7"
Nov 02 09:45:22.328008 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:45:22.327980    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh_odh-ai-gateway-infra(6d745e8e-2ca3-4625-b6a6-e89bb9b49afd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh" podUID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd"
Nov 02 09:45:22.345796 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:22.345772    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"]
Nov 02 09:45:22.490229 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:22.490206    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"
Nov 02 09:45:22.573050 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:22.573019    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c23a6d-c1aa-4fb5-9e57-6faaa004af68-kube-api-access-52jnm\" (UniqueName: \"kubernetes.io/projected/30c23a6d-c1aa-4fb5-9e57-6faaa004af68-kube-api-access-52jnm\") pod \"30c23a6d-c1aa-4fb5-9e57-6faaa004af68\" (UID: \"30c23a6d-c1aa-4fb5-9e57-6faaa004af68\") "
Nov 02 09:45:22.573217 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:22.573161    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c23a6d-c1aa-4fb5-9e57-6faaa004af68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c23a6d-c1aa-4fb5-9e57-6faaa004af68-maas-api-service-ca\") pod \"30c23a6d-c1aa-4fb5-9e57-6faaa004af68\" (UID: \"30c23a6d-c1aa-4fb5-9e57-6faaa004af68\") "
Nov 02 09:45:22.573510 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:22.573489    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c23a6d-c1aa-4fb5-9e57-6faaa004af68-maas-api-service-ca" pod "30c23a6d-c1aa-4fb5-9e57-6faaa004af68" (UID: "30c23a6d-c1aa-4fb5-9e57-6faaa004af68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:45:22.575172 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:22.575150    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c23a6d-c1aa-4fb5-9e57-6faaa004af68-kube-api-access-52jnm" pod "30c23a6d-c1aa-4fb5-9e57-6faaa004af68" (UID: "30c23a6d-c1aa-4fb5-9e57-6faaa004af68"). InnerVolumeSpecName "kube-api-access-52jnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:45:22.674848 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:22.674770    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c23a6d-c1aa-4fb5-9e57-6faaa004af68-maas-api-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:22.674848 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:22.674799    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52jnm\" (UniqueName: \"kubernetes.io/projected/30c23a6d-c1aa-4fb5-9e57-6faaa004af68-kube-api-access-52jnm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:23.176211 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176178    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"]
Nov 02 09:45:23.176705 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176682    4614 cpu_manager.go:414] "containerMap: removing container" podUID="415fabd5-bf72-45b4-b510-5e473469f6ce" containerName="cleanup"
Nov 02 09:45:23.176705 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176701    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415fabd5-bf72-45b4-b510-5e473469f6ce" containerName="cleanup"
Nov 02 09:45:23.176863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176710    4614 cpu_manager.go:414] "containerMap: removing container" podUID="415fabd5-bf72-45b4-b510-5e473469f6ce" containerName="cleanup"
Nov 02 09:45:23.176863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176716    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415fabd5-bf72-45b4-b510-5e473469f6ce" containerName="cleanup"
Nov 02 09:45:23.176863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176727    4614 cpu_manager.go:414] "containerMap: removing container" podUID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68" containerName="cleanup"
Nov 02 09:45:23.176863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176731    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68" containerName="cleanup"
Nov 02 09:45:23.176863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176743    4614 cpu_manager.go:414] "containerMap: removing container" podUID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68" containerName="cleanup"
Nov 02 09:45:23.176863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176751    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68" containerName="cleanup"
Nov 02 09:45:23.176863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176771    4614 cpu_manager.go:414] "containerMap: removing container" podUID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68" containerName="cleanup"
Nov 02 09:45:23.176863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176776    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68" containerName="cleanup"
Nov 02 09:45:23.176863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176838    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68" containerName="cleanup"
Nov 02 09:45:23.176863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176847    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68" containerName="cleanup"
Nov 02 09:45:23.176863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176857    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="415fabd5-bf72-45b4-b510-5e473469f6ce" containerName="cleanup"
Nov 02 09:45:23.176863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176864    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="415fabd5-bf72-45b4-b510-5e473469f6ce" containerName="cleanup"
Nov 02 09:45:23.177219 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176870    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="415fabd5-bf72-45b4-b510-5e473469f6ce" containerName="cleanup"
Nov 02 09:45:23.177219 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.176876    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68" containerName="cleanup"
Nov 02 09:45:23.180241 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.180224    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.184568 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.184550    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-000ffb-openshift-default-dockercfg-hz5bc\""
Nov 02 09:45:23.191861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.191834    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"]
Nov 02 09:45:23.280814 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.280780    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-credential-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.280970 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.280828    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-data\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.280970 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.280860    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-podinfo\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.280970 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.280944    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-workload-certs\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.281107 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.280975    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065a3fb3-353d-4f77-868b-83e98ace1aca-istiod-ca-cert\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.281107 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.281007    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cpzj\" (UniqueName: \"kubernetes.io/projected/065a3fb3-353d-4f77-868b-83e98ace1aca-kube-api-access-7cpzj\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.281178 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.281122    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-workload-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.281178 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.281142    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-envoy\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.281244 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.281176    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-token\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.347426 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.347380    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446" event={"ID":"30c23a6d-c1aa-4fb5-9e57-6faaa004af68","Type":"ContainerDied","Data":"4709188e7068a294294168a7bad252ab324798ebfa763c2083e2250e0e506147"}
Nov 02 09:45:23.347869 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.347678    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"
Nov 02 09:45:23.347869 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.347788    4614 scope.go:122] "RemoveContainer" containerID="bc5b9e3609f2dd86b182eaafcf4964781d3b62c4d156de58c31f9b548cf63ffe"
Nov 02 09:45:23.364001 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.363975    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"]
Nov 02 09:45:23.377393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.377367    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"]
Nov 02 09:45:23.381344 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.381318    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893545-5k446"]
Nov 02 09:45:23.381744 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.381723    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-workload-certs\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.381819 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.381754    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065a3fb3-353d-4f77-868b-83e98ace1aca-istiod-ca-cert\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.381819 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.381798    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cpzj\" (UniqueName: \"kubernetes.io/projected/065a3fb3-353d-4f77-868b-83e98ace1aca-kube-api-access-7cpzj\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.381901 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.381855    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-workload-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.381901 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.381883    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-envoy\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.382710 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.381904    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-token\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.382710 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.382150    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-workload-certs\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.382710 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.382242    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-credential-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.382710 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.382293    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-data\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.382710 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.382330    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-podinfo\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.382710 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.382605    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065a3fb3-353d-4f77-868b-83e98ace1aca-istiod-ca-cert\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.382710 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.382693    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-credential-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.383097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.382983    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-workload-socket\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.383097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.383001    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-data\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.384686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.384658    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-envoy\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.386292 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.386263    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-podinfo\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.402224 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.402190    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-token\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.403114 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.402438    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cpzj\" (UniqueName: \"kubernetes.io/projected/065a3fb3-353d-4f77-868b-83e98ace1aca-kube-api-access-7cpzj\") pod \"e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") " pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.495485 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.495459    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:23.496701 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.496684    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"
Nov 02 09:45:23.584723 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.584693    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd-maas-api-service-ca\") pod \"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd\" (UID: \"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd\") "
Nov 02 09:45:23.584858 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.584729    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd-kube-api-access-tdtbr\" (UniqueName: \"kubernetes.io/projected/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd-kube-api-access-tdtbr\") pod \"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd\" (UID: \"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd\") "
Nov 02 09:45:23.585258 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.585219    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd-maas-api-service-ca" pod "6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" (UID: "6d745e8e-2ca3-4625-b6a6-e89bb9b49afd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:45:23.587273 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.587251    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd-kube-api-access-tdtbr" pod "6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" (UID: "6d745e8e-2ca3-4625-b6a6-e89bb9b49afd"). InnerVolumeSpecName "kube-api-access-tdtbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:45:23.686566 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.686528    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd-maas-api-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:23.686566 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.686561    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdtbr\" (UniqueName: \"kubernetes.io/projected/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd-kube-api-access-tdtbr\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:23.782851 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.782825    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"]
Nov 02 09:45:23.784565 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:45:23.784538    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065a3fb3_353d_4f77_868b_83e98ace1aca.slice/crio-96d43e0782023e61556e131242010a6e098d88e8bf0ba29eebce2af81701fbc2 WatchSource:0}: Error finding container 96d43e0782023e61556e131242010a6e098d88e8bf0ba29eebce2af81701fbc2: Status 404 returned error can't find the container with id 96d43e0782023e61556e131242010a6e098d88e8bf0ba29eebce2af81701fbc2
Nov 02 09:45:23.787141 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.787109    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:45:23.787243 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.787187    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:45:23.787243 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:23.787231    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:45:24.367340 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:24.364219    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh" event={"ID":"6d745e8e-2ca3-4625-b6a6-e89bb9b49afd","Type":"ContainerDied","Data":"449adadd0a939a949088eb588cb599940b5737fddc113a3fee499e21ac4b8ed3"}
Nov 02 09:45:24.367340 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:24.364292    4614 scope.go:122] "RemoveContainer" containerID="5c4f68c17f8c722ef1edcae5afc841286f14ae6fe466706e91e017f4c24e4ac7"
Nov 02 09:45:24.367340 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:24.364410    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"
Nov 02 09:45:24.383686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:24.382732    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf" event={"ID":"065a3fb3-353d-4f77-868b-83e98ace1aca","Type":"ContainerStarted","Data":"470a4f0ca37814600c6156b541fa3e1ac8989344d552f29806cb086ef03045e8"}
Nov 02 09:45:24.383686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:24.382802    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf" event={"ID":"065a3fb3-353d-4f77-868b-83e98ace1aca","Type":"ContainerStarted","Data":"96d43e0782023e61556e131242010a6e098d88e8bf0ba29eebce2af81701fbc2"}
Nov 02 09:45:24.424317 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:24.424266    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf" podStartSLOduration=1.424252232 podStartE2EDuration="1.424252232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:24.422521343 +0000 UTC m=+605.255179424" watchObservedRunningTime="2026-11-02 09:45:24.424252232 +0000 UTC m=+605.256910311"
Nov 02 09:45:24.446375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:24.439764    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"]
Nov 02 09:45:24.450371 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:24.446604    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-000ffb-29893545-rltsh"]
Nov 02 09:45:24.496501 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:24.496467    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:24.497984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:24.497959    4614 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 09:45:24.498093 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:24.498006    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf" podUID="065a3fb3-353d-4f77-868b-83e98ace1aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 09:45:25.318455 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:25.318416    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c23a6d-c1aa-4fb5-9e57-6faaa004af68" path="/var/lib/kubelet/pods/30c23a6d-c1aa-4fb5-9e57-6faaa004af68/volumes"
Nov 02 09:45:25.318866 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:25.318849    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" path="/var/lib/kubelet/pods/6d745e8e-2ca3-4625-b6a6-e89bb9b49afd/volumes"
Nov 02 09:45:25.496510 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:25.496475    4614 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 09:45:25.496879 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:25.496530    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf" podUID="065a3fb3-353d-4f77-868b-83e98ace1aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 09:45:26.496258 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:26.496224    4614 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 09:45:26.496493 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:26.496278    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf" podUID="065a3fb3-353d-4f77-868b-83e98ace1aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 09:45:27.496501 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:27.496463    4614 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 09:45:27.496905 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:27.496516    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf" podUID="065a3fb3-353d-4f77-868b-83e98ace1aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 09:45:28.496095 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:28.496056    4614 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 09:45:28.496266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:28.496127    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf" podUID="065a3fb3-353d-4f77-868b-83e98ace1aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 09:45:29.496807 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:29.496777    4614 patch_prober.go:29] interesting pod/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 09:45:29.497205 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:29.496828    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf" podUID="065a3fb3-353d-4f77-868b-83e98ace1aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 09:45:30.501372 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:30.501307    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:30.501883 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:30.501689    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:30.502559 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:30.502538    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:45:35.852291 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.852246    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"]
Nov 02 09:45:35.853716 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.853692    4614 cpu_manager.go:414] "containerMap: removing container" podUID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" containerName="cleanup"
Nov 02 09:45:35.853716 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.853714    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" containerName="cleanup"
Nov 02 09:45:35.853876 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.853733    4614 cpu_manager.go:414] "containerMap: removing container" podUID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" containerName="cleanup"
Nov 02 09:45:35.853876 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.853738    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" containerName="cleanup"
Nov 02 09:45:35.853876 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.853777    4614 cpu_manager.go:414] "containerMap: removing container" podUID="415fabd5-bf72-45b4-b510-5e473469f6ce" containerName="cleanup"
Nov 02 09:45:35.853876 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.853782    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415fabd5-bf72-45b4-b510-5e473469f6ce" containerName="cleanup"
Nov 02 09:45:35.853876 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.853849    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" containerName="cleanup"
Nov 02 09:45:35.853876 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.853859    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" containerName="cleanup"
Nov 02 09:45:35.853876 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.853865    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" containerName="cleanup"
Nov 02 09:45:35.864634 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.864609    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:35.871508 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.871483    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"]
Nov 02 09:45:35.875455 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.873999    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-bde944\"/\"openshift-service-ca.crt\""
Nov 02 09:45:35.875633 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.874604    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-bde944\"/\"kube-root-ca.crt\""
Nov 02 09:45:35.892545 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.892514    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-bde944\"/\"auth-test-model-bde944-kserve-self-signed-certs\""
Nov 02 09:45:35.892685 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.892616    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-bde944\"/\"default-dockercfg-4ktdd\""
Nov 02 09:45:35.951777 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.951728    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"]
Nov 02 09:45:35.952367 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.952328    4614 cpu_manager.go:414] "containerMap: removing container" podUID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" containerName="cleanup"
Nov 02 09:45:35.952497 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.952368    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d745e8e-2ca3-4625-b6a6-e89bb9b49afd" containerName="cleanup"
Nov 02 09:45:35.955860 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.955837    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"
Nov 02 09:45:35.965529 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.965497    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:45:35.965702 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.965536    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:45:35.966679 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:35.966658    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"]
Nov 02 09:45:36.057389 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.055617    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-model-cache\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.057389 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.055689    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-dshm\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.057389 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.055734    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxjfd\" (UniqueName: \"kubernetes.io/projected/57cde931-be21-4545-bd3c-df85b7170196-kube-api-access-lxjfd\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.057389 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.055769    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b708f2bf-jtfbj\" (UID: \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"
Nov 02 09:45:36.057389 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.055830    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-tmp-dir\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.057389 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.055854    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-home\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.057389 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.055878    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztwlv\" (UniqueName: \"kubernetes.io/projected/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-kube-api-access-ztwlv\") pod \"maas-api-revoke-sub-b708f2bf-jtfbj\" (UID: \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"
Nov 02 09:45:36.057389 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.055922    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b708f2bf-jtfbj\" (UID: \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"
Nov 02 09:45:36.057389 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.056008    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cde931-be21-4545-bd3c-df85b7170196-tls-certs\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.157558 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.157482    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cde931-be21-4545-bd3c-df85b7170196-tls-certs\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.157558 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.157533    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-model-cache\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.157558 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.157556    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-dshm\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.157827 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.157580    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxjfd\" (UniqueName: \"kubernetes.io/projected/57cde931-be21-4545-bd3c-df85b7170196-kube-api-access-lxjfd\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.157827 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.157599    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b708f2bf-jtfbj\" (UID: \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"
Nov 02 09:45:36.157827 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.157776    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-tmp-dir\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.157827 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.157824    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-home\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.158017 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.157851    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztwlv\" (UniqueName: \"kubernetes.io/projected/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-kube-api-access-ztwlv\") pod \"maas-api-revoke-sub-b708f2bf-jtfbj\" (UID: \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"
Nov 02 09:45:36.158017 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.157903    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b708f2bf-jtfbj\" (UID: \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"
Nov 02 09:45:36.158131 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.158116    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-model-cache\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.158219 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.158196    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-tmp-dir\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.158496 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.158473    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-home\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.158692 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.158673    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b708f2bf-jtfbj\" (UID: \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"
Nov 02 09:45:36.160311 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.160293    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-dshm\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.160441 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.160389    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b708f2bf-jtfbj\" (UID: \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"
Nov 02 09:45:36.160639 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.160621    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cde931-be21-4545-bd3c-df85b7170196-tls-certs\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.169970 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.169948    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxjfd\" (UniqueName: \"kubernetes.io/projected/57cde931-be21-4545-bd3c-df85b7170196-kube-api-access-lxjfd\") pod \"auth-test-model-bde944-kserve-5945994c4f-rhl2r\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") " pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.170224 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.170198    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztwlv\" (UniqueName: \"kubernetes.io/projected/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-kube-api-access-ztwlv\") pod \"maas-api-revoke-sub-b708f2bf-jtfbj\" (UID: \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"
Nov 02 09:45:36.193043 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.193018    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:36.270912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.270869    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"
Nov 02 09:45:36.523246 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.523206    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"]
Nov 02 09:45:36.528149 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:45:36.528124    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57cde931_be21_4545_bd3c_df85b7170196.slice/crio-9ea58ba631f30734e673b7cf6438e79d91110acab593cc56f4431adf47afd9c6 WatchSource:0}: Error finding container 9ea58ba631f30734e673b7cf6438e79d91110acab593cc56f4431adf47afd9c6: Status 404 returned error can't find the container with id 9ea58ba631f30734e673b7cf6438e79d91110acab593cc56f4431adf47afd9c6
Nov 02 09:45:36.590347 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:36.590314    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"]
Nov 02 09:45:36.594707 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:45:36.594665    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2da5ca_78ee_4a03_8e4b_7ee199593b6f.slice/crio-4e87adb7b4699cdf6cf08b6be1e9fdb0fc533545db24ddeb55c098da4067defb WatchSource:0}: Error finding container 4e87adb7b4699cdf6cf08b6be1e9fdb0fc533545db24ddeb55c098da4067defb: Status 404 returned error can't find the container with id 4e87adb7b4699cdf6cf08b6be1e9fdb0fc533545db24ddeb55c098da4067defb
Nov 02 09:45:37.492276 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:45:37.492238    4614 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2da5ca_78ee_4a03_8e4b_7ee199593b6f.slice/crio-conmon-2a3e1eac805bd9f381441622b690fc27bbda672dd7170ac033a400665d019a77.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 09:45:37.497521 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:37.497495    4614 generic.go:356] "Generic (PLEG): container finished" podID="4a2da5ca-78ee-4a03-8e4b-7ee199593b6f" containerID="2a3e1eac805bd9f381441622b690fc27bbda672dd7170ac033a400665d019a77" exitCode=0
Nov 02 09:45:37.497645 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:37.497574    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj" event={"ID":"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f","Type":"ContainerDied","Data":"2a3e1eac805bd9f381441622b690fc27bbda672dd7170ac033a400665d019a77"}
Nov 02 09:45:37.497645 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:37.497632    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj" event={"ID":"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f","Type":"ContainerStarted","Data":"4e87adb7b4699cdf6cf08b6be1e9fdb0fc533545db24ddeb55c098da4067defb"}
Nov 02 09:45:37.499214 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:37.499192    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r" event={"ID":"57cde931-be21-4545-bd3c-df85b7170196","Type":"ContainerStarted","Data":"a76dfafb09a5ca42c060a2a5f115b02dbccf1d7b82b47876f7686c68d9c7b8d6"}
Nov 02 09:45:37.499315 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:37.499225    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r" event={"ID":"57cde931-be21-4545-bd3c-df85b7170196","Type":"ContainerStarted","Data":"9ea58ba631f30734e673b7cf6438e79d91110acab593cc56f4431adf47afd9c6"}
Nov 02 09:45:37.536604 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:37.536558    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r" podStartSLOduration=2.536544036 podStartE2EDuration="2.536544036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:37.534823463 +0000 UTC m=+618.367481544" watchObservedRunningTime="2026-11-02 09:45:37.536544036 +0000 UTC m=+618.369202165"
Nov 02 09:45:38.193406 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:38.193337    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:38.657503 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:38.657480    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"
Nov 02 09:45:38.685929 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:38.685904    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-kube-api-access-ztwlv\" (UniqueName: \"kubernetes.io/projected/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-kube-api-access-ztwlv\") pod \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\" (UID: \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\") "
Nov 02 09:45:38.686090 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:38.685973    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-maas-api-xks-ca\") pod \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\" (UID: \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\") "
Nov 02 09:45:38.686166 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:38.686090    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-maas-api-openshift-ca\") pod \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\" (UID: \"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f\") "
Nov 02 09:45:38.686510 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:38.686477    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-maas-api-openshift-ca" pod "4a2da5ca-78ee-4a03-8e4b-7ee199593b6f" (UID: "4a2da5ca-78ee-4a03-8e4b-7ee199593b6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:45:38.688033 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:38.688006    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-maas-api-xks-ca" pod "4a2da5ca-78ee-4a03-8e4b-7ee199593b6f" (UID: "4a2da5ca-78ee-4a03-8e4b-7ee199593b6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:45:38.688473 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:38.688452    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-kube-api-access-ztwlv" pod "4a2da5ca-78ee-4a03-8e4b-7ee199593b6f" (UID: "4a2da5ca-78ee-4a03-8e4b-7ee199593b6f"). InnerVolumeSpecName "kube-api-access-ztwlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:45:38.787062 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:38.786991    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztwlv\" (UniqueName: \"kubernetes.io/projected/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-kube-api-access-ztwlv\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:38.787062 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:38.787018    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:38.787062 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:38.787029    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:39.511413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:39.511380    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"
Nov 02 09:45:39.511413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:39.511394    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj" event={"ID":"4a2da5ca-78ee-4a03-8e4b-7ee199593b6f","Type":"ContainerDied","Data":"4e87adb7b4699cdf6cf08b6be1e9fdb0fc533545db24ddeb55c098da4067defb"}
Nov 02 09:45:39.511629 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:39.511440    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e87adb7b4699cdf6cf08b6be1e9fdb0fc533545db24ddeb55c098da4067defb"
Nov 02 09:45:39.635865 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:45:39.635775    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:45:44.929372 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:44.929321    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"]
Nov 02 09:45:46.193322 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:46.193290    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:46.210894 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:46.210863    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:46.562888 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:46.562860    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:45:51.529868 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.529828    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"]
Nov 02 09:45:51.530380 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.530361    4614 cpu_manager.go:414] "containerMap: removing container" podUID="4a2da5ca-78ee-4a03-8e4b-7ee199593b6f" containerName="revoke-keys"
Nov 02 09:45:51.530380 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.530378    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2da5ca-78ee-4a03-8e4b-7ee199593b6f" containerName="revoke-keys"
Nov 02 09:45:51.530557 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.530462    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2da5ca-78ee-4a03-8e4b-7ee199593b6f" containerName="revoke-keys"
Nov 02 09:45:51.533925 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.533907    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"
Nov 02 09:45:51.538700 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.538682    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:45:51.540282 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.540261    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:45:51.546677 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.546652    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"]
Nov 02 09:45:51.641712 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.641679    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12148caa-3e3f-471d-a0f8-572e58e35b7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4d50b8-dvbj5\" (UID: \"12148caa-3e3f-471d-a0f8-572e58e35b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"
Nov 02 09:45:51.641878 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.641795    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44kpm\" (UniqueName: \"kubernetes.io/projected/12148caa-3e3f-471d-a0f8-572e58e35b7b-kube-api-access-44kpm\") pod \"maas-api-revoke-sub-8d4d50b8-dvbj5\" (UID: \"12148caa-3e3f-471d-a0f8-572e58e35b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"
Nov 02 09:45:51.641878 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.641845    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12148caa-3e3f-471d-a0f8-572e58e35b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4d50b8-dvbj5\" (UID: \"12148caa-3e3f-471d-a0f8-572e58e35b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"
Nov 02 09:45:51.743136 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.743099    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12148caa-3e3f-471d-a0f8-572e58e35b7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4d50b8-dvbj5\" (UID: \"12148caa-3e3f-471d-a0f8-572e58e35b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"
Nov 02 09:45:51.743314 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.743154    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44kpm\" (UniqueName: \"kubernetes.io/projected/12148caa-3e3f-471d-a0f8-572e58e35b7b-kube-api-access-44kpm\") pod \"maas-api-revoke-sub-8d4d50b8-dvbj5\" (UID: \"12148caa-3e3f-471d-a0f8-572e58e35b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"
Nov 02 09:45:51.743392 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.743321    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12148caa-3e3f-471d-a0f8-572e58e35b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4d50b8-dvbj5\" (UID: \"12148caa-3e3f-471d-a0f8-572e58e35b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"
Nov 02 09:45:51.743984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.743959    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12148caa-3e3f-471d-a0f8-572e58e35b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4d50b8-dvbj5\" (UID: \"12148caa-3e3f-471d-a0f8-572e58e35b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"
Nov 02 09:45:51.745524 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.745505    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12148caa-3e3f-471d-a0f8-572e58e35b7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4d50b8-dvbj5\" (UID: \"12148caa-3e3f-471d-a0f8-572e58e35b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"
Nov 02 09:45:51.756130 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.756104    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44kpm\" (UniqueName: \"kubernetes.io/projected/12148caa-3e3f-471d-a0f8-572e58e35b7b-kube-api-access-44kpm\") pod \"maas-api-revoke-sub-8d4d50b8-dvbj5\" (UID: \"12148caa-3e3f-471d-a0f8-572e58e35b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"
Nov 02 09:45:51.863984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:51.863913    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"
Nov 02 09:45:52.165270 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.165247    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"]
Nov 02 09:45:52.167263 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:45:52.167234    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12148caa_3e3f_471d_a0f8_572e58e35b7b.slice/crio-2d0ff76188c0645b481e5cc1ced1e5e2bcc5fb054d61924b8f8eabf1bb7388cd WatchSource:0}: Error finding container 2d0ff76188c0645b481e5cc1ced1e5e2bcc5fb054d61924b8f8eabf1bb7388cd: Status 404 returned error can't find the container with id 2d0ff76188c0645b481e5cc1ced1e5e2bcc5fb054d61924b8f8eabf1bb7388cd
Nov 02 09:45:52.582157 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.582111    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5" event={"ID":"12148caa-3e3f-471d-a0f8-572e58e35b7b","Type":"ContainerStarted","Data":"1cdff5713a3690c91319ef1450a35eb96c6500fbcc0192cbc09c4421c2596c7c"}
Nov 02 09:45:52.582743 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.582169    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5" event={"ID":"12148caa-3e3f-471d-a0f8-572e58e35b7b","Type":"ContainerStarted","Data":"2d0ff76188c0645b481e5cc1ced1e5e2bcc5fb054d61924b8f8eabf1bb7388cd"}
Nov 02 09:45:52.602192 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.602144    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5" podStartSLOduration=1.602130587 podStartE2EDuration="1.602130587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:52.600802468 +0000 UTC m=+633.433460542" watchObservedRunningTime="2026-11-02 09:45:52.602130587 +0000 UTC m=+633.434788692"
Nov 02 09:45:52.783429 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.783397    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"]
Nov 02 09:45:52.787426 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.787403    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.796746 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.796721    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9e7156\"/\"auth-test-model-9e7156-kserve-self-signed-certs\""
Nov 02 09:45:52.796910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.796829    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9e7156\"/\"kube-root-ca.crt\""
Nov 02 09:45:52.796910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.796849    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9e7156\"/\"default-dockercfg-8h4tz\""
Nov 02 09:45:52.796910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.796892    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9e7156\"/\"openshift-service-ca.crt\""
Nov 02 09:45:52.799772 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.799747    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"]
Nov 02 09:45:52.853975 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.853944    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-dshm\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.854145 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.854002    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-tls-certs\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.854145 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.854068    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4j58\" (UniqueName: \"kubernetes.io/projected/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-kube-api-access-m4j58\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.854217 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.854144    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-tmp-dir\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.854251 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.854221    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-home\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.854284 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.854275    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-model-cache\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.957013 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.956970    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-model-cache\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.957823 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.957791    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-dshm\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.957974 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.957853    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-tls-certs\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.957974 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.957879    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j58\" (UniqueName: \"kubernetes.io/projected/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-kube-api-access-m4j58\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.957974 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.957903    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-tmp-dir\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.958155 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.958025    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-home\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.962374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.958380    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-home\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.962374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.957693    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-model-cache\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.962374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.961866    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-tmp-dir\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.981413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.979699    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-dshm\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.981413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.980959    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4j58\" (UniqueName: \"kubernetes.io/projected/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-kube-api-access-m4j58\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:52.981413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:52.981338    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-tls-certs\") pod \"auth-test-model-9e7156-kserve-7f77bf9477-sl6fl\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") " pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:53.103451 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:53.103421    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:53.387652 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:53.387631    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"]
Nov 02 09:45:53.389920 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:45:53.389891    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod623fd585_5bd4_4a62_87fb_6c27d59bb0c0.slice/crio-76b5598e0e8746d7e7557c485a859d5224de2863291cc85c724a81726ebffe78 WatchSource:0}: Error finding container 76b5598e0e8746d7e7557c485a859d5224de2863291cc85c724a81726ebffe78: Status 404 returned error can't find the container with id 76b5598e0e8746d7e7557c485a859d5224de2863291cc85c724a81726ebffe78
Nov 02 09:45:53.591287 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:53.590986    4614 generic.go:356] "Generic (PLEG): container finished" podID="12148caa-3e3f-471d-a0f8-572e58e35b7b" containerID="1cdff5713a3690c91319ef1450a35eb96c6500fbcc0192cbc09c4421c2596c7c" exitCode=0
Nov 02 09:45:53.591287 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:53.591217    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5" event={"ID":"12148caa-3e3f-471d-a0f8-572e58e35b7b","Type":"ContainerDied","Data":"1cdff5713a3690c91319ef1450a35eb96c6500fbcc0192cbc09c4421c2596c7c"}
Nov 02 09:45:53.593935 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:53.593906    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl" event={"ID":"623fd585-5bd4-4a62-87fb-6c27d59bb0c0","Type":"ContainerStarted","Data":"a4fc390a694c6f0414fdf77ea82d29f4dd896eacfe50d836bd3c34ccd2e45727"}
Nov 02 09:45:53.594138 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:53.594119    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl" event={"ID":"623fd585-5bd4-4a62-87fb-6c27d59bb0c0","Type":"ContainerStarted","Data":"76b5598e0e8746d7e7557c485a859d5224de2863291cc85c724a81726ebffe78"}
Nov 02 09:45:53.636500 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:53.636440    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl" podStartSLOduration=1.636419864 podStartE2EDuration="1.636419864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:45:53.634677693 +0000 UTC m=+634.467335774" watchObservedRunningTime="2026-11-02 09:45:53.636419864 +0000 UTC m=+634.469077944"
Nov 02 09:45:54.104524 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:54.104487    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:45:54.792090 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:54.792053    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"
Nov 02 09:45:54.883916 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:54.883838    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12148caa-3e3f-471d-a0f8-572e58e35b7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12148caa-3e3f-471d-a0f8-572e58e35b7b-maas-api-openshift-ca\") pod \"12148caa-3e3f-471d-a0f8-572e58e35b7b\" (UID: \"12148caa-3e3f-471d-a0f8-572e58e35b7b\") "
Nov 02 09:45:54.884079 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:54.883990    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12148caa-3e3f-471d-a0f8-572e58e35b7b-kube-api-access-44kpm\" (UniqueName: \"kubernetes.io/projected/12148caa-3e3f-471d-a0f8-572e58e35b7b-kube-api-access-44kpm\") pod \"12148caa-3e3f-471d-a0f8-572e58e35b7b\" (UID: \"12148caa-3e3f-471d-a0f8-572e58e35b7b\") "
Nov 02 09:45:54.884079 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:54.884063    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12148caa-3e3f-471d-a0f8-572e58e35b7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12148caa-3e3f-471d-a0f8-572e58e35b7b-maas-api-xks-ca\") pod \"12148caa-3e3f-471d-a0f8-572e58e35b7b\" (UID: \"12148caa-3e3f-471d-a0f8-572e58e35b7b\") "
Nov 02 09:45:54.884329 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:54.884300    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12148caa-3e3f-471d-a0f8-572e58e35b7b-maas-api-openshift-ca" pod "12148caa-3e3f-471d-a0f8-572e58e35b7b" (UID: "12148caa-3e3f-471d-a0f8-572e58e35b7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:45:54.886152 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:54.886122    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12148caa-3e3f-471d-a0f8-572e58e35b7b-kube-api-access-44kpm" pod "12148caa-3e3f-471d-a0f8-572e58e35b7b" (UID: "12148caa-3e3f-471d-a0f8-572e58e35b7b"). InnerVolumeSpecName "kube-api-access-44kpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:45:54.886263 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:54.886212    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12148caa-3e3f-471d-a0f8-572e58e35b7b-maas-api-xks-ca" pod "12148caa-3e3f-471d-a0f8-572e58e35b7b" (UID: "12148caa-3e3f-471d-a0f8-572e58e35b7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:45:54.985810 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:54.985781    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44kpm\" (UniqueName: \"kubernetes.io/projected/12148caa-3e3f-471d-a0f8-572e58e35b7b-kube-api-access-44kpm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:54.985810 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:54.985809    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12148caa-3e3f-471d-a0f8-572e58e35b7b-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:54.985996 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:54.985823    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12148caa-3e3f-471d-a0f8-572e58e35b7b-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:45:55.608097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:55.608025    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"
Nov 02 09:45:55.608097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:55.608057    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5" event={"ID":"12148caa-3e3f-471d-a0f8-572e58e35b7b","Type":"ContainerDied","Data":"2d0ff76188c0645b481e5cc1ced1e5e2bcc5fb054d61924b8f8eabf1bb7388cd"}
Nov 02 09:45:55.608097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:55.608105    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d0ff76188c0645b481e5cc1ced1e5e2bcc5fb054d61924b8f8eabf1bb7388cd"
Nov 02 09:45:59.949818 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:45:59.949779    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw" podUID="f0d14fe7-3f66-4328-a2ef-535f741c1c05" containerName="main" containerID="cri-o://0d0d74945286d9d02f505015b161f35c28de37752e637990882bd9448d78b5d2" gracePeriod=45
Nov 02 09:46:00.211136 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.211082    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:46:00.350710 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.350681    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-tmp-dir\") pod \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") "
Nov 02 09:46:00.350885 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.350751    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d14fe7-3f66-4328-a2ef-535f741c1c05-kube-api-access-8zmp2\" (UniqueName: \"kubernetes.io/projected/f0d14fe7-3f66-4328-a2ef-535f741c1c05-kube-api-access-8zmp2\") pod \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") "
Nov 02 09:46:00.350885 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.350817    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0d14fe7-3f66-4328-a2ef-535f741c1c05-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d14fe7-3f66-4328-a2ef-535f741c1c05-tls-certs\") pod \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") "
Nov 02 09:46:00.350885 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.350844    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-dshm\") pod \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") "
Nov 02 09:46:00.351003 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.350893    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-model-cache\") pod \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") "
Nov 02 09:46:00.351003 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.350942    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-home\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-home\") pod \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\" (UID: \"f0d14fe7-3f66-4328-a2ef-535f741c1c05\") "
Nov 02 09:46:00.351115 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.351008    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-tmp-dir" pod "f0d14fe7-3f66-4328-a2ef-535f741c1c05" (UID: "f0d14fe7-3f66-4328-a2ef-535f741c1c05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:00.351264 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.351233    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-model-cache" pod "f0d14fe7-3f66-4328-a2ef-535f741c1c05" (UID: "f0d14fe7-3f66-4328-a2ef-535f741c1c05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:00.351540 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.351486    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-home" pod "f0d14fe7-3f66-4328-a2ef-535f741c1c05" (UID: "f0d14fe7-3f66-4328-a2ef-535f741c1c05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:00.351627 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.351565    4614 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-model-cache\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:00.351627 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.351585    4614 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-tmp-dir\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:00.353173 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.353147    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d14fe7-3f66-4328-a2ef-535f741c1c05-kube-api-access-8zmp2" pod "f0d14fe7-3f66-4328-a2ef-535f741c1c05" (UID: "f0d14fe7-3f66-4328-a2ef-535f741c1c05"). InnerVolumeSpecName "kube-api-access-8zmp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:00.353173 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.353166    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-dshm" pod "f0d14fe7-3f66-4328-a2ef-535f741c1c05" (UID: "f0d14fe7-3f66-4328-a2ef-535f741c1c05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:00.353299 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.353176    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d14fe7-3f66-4328-a2ef-535f741c1c05-tls-certs" pod "f0d14fe7-3f66-4328-a2ef-535f741c1c05" (UID: "f0d14fe7-3f66-4328-a2ef-535f741c1c05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:46:00.452466 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.452433    4614 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-home\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:00.452466 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.452460    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zmp2\" (UniqueName: \"kubernetes.io/projected/f0d14fe7-3f66-4328-a2ef-535f741c1c05-kube-api-access-8zmp2\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:00.452466 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.452470    4614 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d14fe7-3f66-4328-a2ef-535f741c1c05-tls-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:00.452699 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.452479    4614 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d14fe7-3f66-4328-a2ef-535f741c1c05-dshm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:00.636227 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.636190    4614 generic.go:356] "Generic (PLEG): container finished" podID="f0d14fe7-3f66-4328-a2ef-535f741c1c05" containerID="0d0d74945286d9d02f505015b161f35c28de37752e637990882bd9448d78b5d2" exitCode=0
Nov 02 09:46:00.636451 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.636243    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"
Nov 02 09:46:00.636451 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.636283    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw" event={"ID":"f0d14fe7-3f66-4328-a2ef-535f741c1c05","Type":"ContainerDied","Data":"0d0d74945286d9d02f505015b161f35c28de37752e637990882bd9448d78b5d2"}
Nov 02 09:46:00.636451 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.636328    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw" event={"ID":"f0d14fe7-3f66-4328-a2ef-535f741c1c05","Type":"ContainerDied","Data":"3950e36b9813eb03c8732bc5106380031378a6c557383744d37a36ddce574707"}
Nov 02 09:46:00.636451 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.636366    4614 scope.go:122] "RemoveContainer" containerID="0d0d74945286d9d02f505015b161f35c28de37752e637990882bd9448d78b5d2"
Nov 02 09:46:00.655949 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.655522    4614 scope.go:122] "RemoveContainer" containerID="0d0d74945286d9d02f505015b161f35c28de37752e637990882bd9448d78b5d2"
Nov 02 09:46:00.656381 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:46:00.656324    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d0d74945286d9d02f505015b161f35c28de37752e637990882bd9448d78b5d2\": container with ID starting with 0d0d74945286d9d02f505015b161f35c28de37752e637990882bd9448d78b5d2 not found: ID does not exist" containerID="0d0d74945286d9d02f505015b161f35c28de37752e637990882bd9448d78b5d2"
Nov 02 09:46:00.656488 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.656384    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d0d74945286d9d02f505015b161f35c28de37752e637990882bd9448d78b5d2"} err="failed to get container status \"0d0d74945286d9d02f505015b161f35c28de37752e637990882bd9448d78b5d2\": rpc error: code = NotFound desc = could not find container \"0d0d74945286d9d02f505015b161f35c28de37752e637990882bd9448d78b5d2\": container with ID starting with 0d0d74945286d9d02f505015b161f35c28de37752e637990882bd9448d78b5d2 not found: ID does not exist"
Nov 02 09:46:00.673909 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.673880    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"]
Nov 02 09:46:00.677099 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:00.677077    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-000ffb/praxis-route-000ffb-kserve-5d889b6685-p7ltw"]
Nov 02 09:46:01.315574 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:01.315545    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d14fe7-3f66-4328-a2ef-535f741c1c05" path="/var/lib/kubelet/pods/f0d14fe7-3f66-4328-a2ef-535f741c1c05/volumes"
Nov 02 09:46:03.103851 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:03.103815    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:46:03.117259 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:03.117231    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:46:03.665542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:03.665514    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:46:04.490733 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.490704    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"]
Nov 02 09:46:04.491102 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.491078    4614 cpu_manager.go:414] "containerMap: removing container" podUID="12148caa-3e3f-471d-a0f8-572e58e35b7b" containerName="revoke-keys"
Nov 02 09:46:04.491102 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.491090    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12148caa-3e3f-471d-a0f8-572e58e35b7b" containerName="revoke-keys"
Nov 02 09:46:04.491185 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.491117    4614 cpu_manager.go:414] "containerMap: removing container" podUID="f0d14fe7-3f66-4328-a2ef-535f741c1c05" containerName="main"
Nov 02 09:46:04.491185 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.491123    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d14fe7-3f66-4328-a2ef-535f741c1c05" containerName="main"
Nov 02 09:46:04.491185 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.491177    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="12148caa-3e3f-471d-a0f8-572e58e35b7b" containerName="revoke-keys"
Nov 02 09:46:04.491322 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.491190    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d14fe7-3f66-4328-a2ef-535f741c1c05" containerName="main"
Nov 02 09:46:04.522896 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.522867    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"]
Nov 02 09:46:04.523051 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.522984    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"
Nov 02 09:46:04.527826 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.527805    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:46:04.527826 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.527804    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:46:04.607637 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.607605    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbkmq\" (UniqueName: \"kubernetes.io/projected/2647b34f-b117-4685-82c2-61eeeca4837e-kube-api-access-rbkmq\") pod \"maas-api-revoke-sub-bdcc2d17-kh9hw\" (UID: \"2647b34f-b117-4685-82c2-61eeeca4837e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"
Nov 02 09:46:04.607810 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.607656    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2647b34f-b117-4685-82c2-61eeeca4837e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdcc2d17-kh9hw\" (UID: \"2647b34f-b117-4685-82c2-61eeeca4837e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"
Nov 02 09:46:04.607810 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.607761    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2647b34f-b117-4685-82c2-61eeeca4837e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdcc2d17-kh9hw\" (UID: \"2647b34f-b117-4685-82c2-61eeeca4837e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"
Nov 02 09:46:04.708663 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.708625    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2647b34f-b117-4685-82c2-61eeeca4837e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdcc2d17-kh9hw\" (UID: \"2647b34f-b117-4685-82c2-61eeeca4837e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"
Nov 02 09:46:04.708842 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.708723    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbkmq\" (UniqueName: \"kubernetes.io/projected/2647b34f-b117-4685-82c2-61eeeca4837e-kube-api-access-rbkmq\") pod \"maas-api-revoke-sub-bdcc2d17-kh9hw\" (UID: \"2647b34f-b117-4685-82c2-61eeeca4837e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"
Nov 02 09:46:04.708842 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.708747    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2647b34f-b117-4685-82c2-61eeeca4837e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdcc2d17-kh9hw\" (UID: \"2647b34f-b117-4685-82c2-61eeeca4837e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"
Nov 02 09:46:04.709379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.709329    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2647b34f-b117-4685-82c2-61eeeca4837e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdcc2d17-kh9hw\" (UID: \"2647b34f-b117-4685-82c2-61eeeca4837e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"
Nov 02 09:46:04.711183 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.711154    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2647b34f-b117-4685-82c2-61eeeca4837e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdcc2d17-kh9hw\" (UID: \"2647b34f-b117-4685-82c2-61eeeca4837e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"
Nov 02 09:46:04.721899 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.721876    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbkmq\" (UniqueName: \"kubernetes.io/projected/2647b34f-b117-4685-82c2-61eeeca4837e-kube-api-access-rbkmq\") pod \"maas-api-revoke-sub-bdcc2d17-kh9hw\" (UID: \"2647b34f-b117-4685-82c2-61eeeca4837e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"
Nov 02 09:46:04.864117 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:04.864046    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"
Nov 02 09:46:05.149366 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:05.149322    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"]
Nov 02 09:46:05.151494 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:46:05.151468    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2647b34f_b117_4685_82c2_61eeeca4837e.slice/crio-b51614d1ee4bb6edd0932642b5b6abb93ee89f3845d7b70c675054b73c539085 WatchSource:0}: Error finding container b51614d1ee4bb6edd0932642b5b6abb93ee89f3845d7b70c675054b73c539085: Status 404 returned error can't find the container with id b51614d1ee4bb6edd0932642b5b6abb93ee89f3845d7b70c675054b73c539085
Nov 02 09:46:05.668114 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:05.668078    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw" event={"ID":"2647b34f-b117-4685-82c2-61eeeca4837e","Type":"ContainerStarted","Data":"b43d1d6f3e587f78fcea912e37319647a6f7979e03ffc952be735cca0b044d18"}
Nov 02 09:46:05.668589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:05.668140    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw" event={"ID":"2647b34f-b117-4685-82c2-61eeeca4837e","Type":"ContainerStarted","Data":"b51614d1ee4bb6edd0932642b5b6abb93ee89f3845d7b70c675054b73c539085"}
Nov 02 09:46:05.688695 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:05.688558    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw" podStartSLOduration=1.688541226 podStartE2EDuration="1.688541226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:05.68591428 +0000 UTC m=+646.518572362" watchObservedRunningTime="2026-11-02 09:46:05.688541226 +0000 UTC m=+646.521199307"
Nov 02 09:46:06.674973 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:06.674936    4614 generic.go:356] "Generic (PLEG): container finished" podID="2647b34f-b117-4685-82c2-61eeeca4837e" containerID="b43d1d6f3e587f78fcea912e37319647a6f7979e03ffc952be735cca0b044d18" exitCode=0
Nov 02 09:46:06.675415 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:06.675028    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw" event={"ID":"2647b34f-b117-4685-82c2-61eeeca4837e","Type":"ContainerDied","Data":"b43d1d6f3e587f78fcea912e37319647a6f7979e03ffc952be735cca0b044d18"}
Nov 02 09:46:07.341917 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.341883    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"]
Nov 02 09:46:07.344668 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.344651    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.349709 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.349680    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a110ef-openshift-default-dockercfg-qjmqj\""
Nov 02 09:46:07.359618 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.359593    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"]
Nov 02 09:46:07.435927 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.435885    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.436112 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.435994    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.436112 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.436027    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.436112 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.436083    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.436112 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.436101    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.436323 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.436178    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.436323 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.436233    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.436323 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.436296    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4j7z\" (UniqueName: \"kubernetes.io/projected/ae2706aa-d41a-463a-acf6-79c6b2705f9c-kube-api-access-m4j7z\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.436563 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.436333    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.538086 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.538037    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.538295 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.538097    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j7z\" (UniqueName: \"kubernetes.io/projected/ae2706aa-d41a-463a-acf6-79c6b2705f9c-kube-api-access-m4j7z\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.538295 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.538125    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.538295 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.538207    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.538295 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.538273    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.538553 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.538320    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.538553 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.538399    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.538553 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.538425    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.538553 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.538471    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.539780 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.539664    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.540533 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.540478    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.540918 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.540890    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.541088 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.541055    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.541271 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.541251    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.546015 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.545970    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.550924 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.550897    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.556738 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.556707    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.560898 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.560867    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4j7z\" (UniqueName: \"kubernetes.io/projected/ae2706aa-d41a-463a-acf6-79c6b2705f9c-kube-api-access-m4j7z\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.657927 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.657847    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:07.841247 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.841223    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"
Nov 02 09:46:07.941733 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.941656    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2647b34f-b117-4685-82c2-61eeeca4837e-kube-api-access-rbkmq\" (UniqueName: \"kubernetes.io/projected/2647b34f-b117-4685-82c2-61eeeca4837e-kube-api-access-rbkmq\") pod \"2647b34f-b117-4685-82c2-61eeeca4837e\" (UID: \"2647b34f-b117-4685-82c2-61eeeca4837e\") "
Nov 02 09:46:07.941886 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.941772    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2647b34f-b117-4685-82c2-61eeeca4837e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2647b34f-b117-4685-82c2-61eeeca4837e-maas-api-openshift-ca\") pod \"2647b34f-b117-4685-82c2-61eeeca4837e\" (UID: \"2647b34f-b117-4685-82c2-61eeeca4837e\") "
Nov 02 09:46:07.941886 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.941805    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2647b34f-b117-4685-82c2-61eeeca4837e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2647b34f-b117-4685-82c2-61eeeca4837e-maas-api-xks-ca\") pod \"2647b34f-b117-4685-82c2-61eeeca4837e\" (UID: \"2647b34f-b117-4685-82c2-61eeeca4837e\") "
Nov 02 09:46:07.942169 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.942148    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2647b34f-b117-4685-82c2-61eeeca4837e-maas-api-openshift-ca" pod "2647b34f-b117-4685-82c2-61eeeca4837e" (UID: "2647b34f-b117-4685-82c2-61eeeca4837e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:46:07.943821 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.943796    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2647b34f-b117-4685-82c2-61eeeca4837e-maas-api-xks-ca" pod "2647b34f-b117-4685-82c2-61eeeca4837e" (UID: "2647b34f-b117-4685-82c2-61eeeca4837e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:46:07.943899 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:07.943802    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2647b34f-b117-4685-82c2-61eeeca4837e-kube-api-access-rbkmq" pod "2647b34f-b117-4685-82c2-61eeeca4837e" (UID: "2647b34f-b117-4685-82c2-61eeeca4837e"). InnerVolumeSpecName "kube-api-access-rbkmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:08.042867 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:08.042837    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbkmq\" (UniqueName: \"kubernetes.io/projected/2647b34f-b117-4685-82c2-61eeeca4837e-kube-api-access-rbkmq\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:08.042867 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:08.042862    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2647b34f-b117-4685-82c2-61eeeca4837e-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:08.042867 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:08.042872    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2647b34f-b117-4685-82c2-61eeeca4837e-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:08.058963 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:08.058937    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"]
Nov 02 09:46:08.077961 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:46:08.077934    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2706aa_d41a_463a_acf6_79c6b2705f9c.slice/crio-ccb2a73cc7479c43b9c786d8b93792782506aa4778b6078a8942cd18feac59bb WatchSource:0}: Error finding container ccb2a73cc7479c43b9c786d8b93792782506aa4778b6078a8942cd18feac59bb: Status 404 returned error can't find the container with id ccb2a73cc7479c43b9c786d8b93792782506aa4778b6078a8942cd18feac59bb
Nov 02 09:46:08.080291 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:08.080262    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:46:08.080382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:08.080327    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:46:08.080382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:08.080370    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:46:08.696057 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:08.696014    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492" event={"ID":"ae2706aa-d41a-463a-acf6-79c6b2705f9c","Type":"ContainerStarted","Data":"48f208575364f300d286b90bb0c7622665ffb21b9a3df7fc159bbad69c181246"}
Nov 02 09:46:08.696260 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:08.696136    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492" event={"ID":"ae2706aa-d41a-463a-acf6-79c6b2705f9c","Type":"ContainerStarted","Data":"ccb2a73cc7479c43b9c786d8b93792782506aa4778b6078a8942cd18feac59bb"}
Nov 02 09:46:08.697971 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:08.697946    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"
Nov 02 09:46:08.697971 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:08.697963    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw" event={"ID":"2647b34f-b117-4685-82c2-61eeeca4837e","Type":"ContainerDied","Data":"b51614d1ee4bb6edd0932642b5b6abb93ee89f3845d7b70c675054b73c539085"}
Nov 02 09:46:08.698152 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:08.698002    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b51614d1ee4bb6edd0932642b5b6abb93ee89f3845d7b70c675054b73c539085"
Nov 02 09:46:08.720875 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:08.720830    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492" podStartSLOduration=1.7208194209999998 podStartE2EDuration="1.720819421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:08.719719533 +0000 UTC m=+649.552377641" watchObservedRunningTime="2026-11-02 09:46:08.720819421 +0000 UTC m=+649.553477501"
Nov 02 09:46:09.658847 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:09.658804    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:09.659875 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:09.659848    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 02 09:46:09.659950 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:09.659905    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492" podUID="ae2706aa-d41a-463a-acf6-79c6b2705f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 02 09:46:09.661600 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:46:09.661561    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:46:10.658621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:10.658587    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 02 09:46:10.658811 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:10.658647    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492" podUID="ae2706aa-d41a-463a-acf6-79c6b2705f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 02 09:46:11.658166 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:11.658134    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 02 09:46:11.658656 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:11.658204    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492" podUID="ae2706aa-d41a-463a-acf6-79c6b2705f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 02 09:46:12.658210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:12.658169    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 02 09:46:12.658633 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:12.658236    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492" podUID="ae2706aa-d41a-463a-acf6-79c6b2705f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 02 09:46:13.661731 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:13.661695    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 02 09:46:13.662369 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:13.662318    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492" podUID="ae2706aa-d41a-463a-acf6-79c6b2705f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 02 09:46:14.373517 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.373482    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"]
Nov 02 09:46:14.374154 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.374135    4614 cpu_manager.go:414] "containerMap: removing container" podUID="2647b34f-b117-4685-82c2-61eeeca4837e" containerName="revoke-keys"
Nov 02 09:46:14.374219 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.374157    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2647b34f-b117-4685-82c2-61eeeca4837e" containerName="revoke-keys"
Nov 02 09:46:14.374294 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.374282    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="2647b34f-b117-4685-82c2-61eeeca4837e" containerName="revoke-keys"
Nov 02 09:46:14.377618 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.377590    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.390722 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.390698    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a110ef\"/\"distinct-w3-a110ef-kserve-self-signed-certs\""
Nov 02 09:46:14.391553 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.391532    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"]
Nov 02 09:46:14.404309 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.404285    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-tmp-dir\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.404461 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.404316    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnbq6\" (UniqueName: \"kubernetes.io/projected/a21c87af-b381-44b0-84f8-0f7b15db29d6-kube-api-access-wnbq6\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.404461 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.404338    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21c87af-b381-44b0-84f8-0f7b15db29d6-tls-certs\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.404461 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.404425    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-home\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.404628 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.404564    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-dshm\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.404685 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.404627    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-model-cache\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.505149 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.505112    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21c87af-b381-44b0-84f8-0f7b15db29d6-tls-certs\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.505365 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.505174    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-home\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.505365 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.505239    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-dshm\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.505365 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.505292    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-model-cache\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.505365 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.505329    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-tmp-dir\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.505365 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.505346    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnbq6\" (UniqueName: \"kubernetes.io/projected/a21c87af-b381-44b0-84f8-0f7b15db29d6-kube-api-access-wnbq6\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.506003 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.505976    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-model-cache\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.506262 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.506239    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-tmp-dir\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.506532 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.506513    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-home\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.529387 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.527330    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-dshm\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.529387 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.527984    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21c87af-b381-44b0-84f8-0f7b15db29d6-tls-certs\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.556386 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.553248    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnbq6\" (UniqueName: \"kubernetes.io/projected/a21c87af-b381-44b0-84f8-0f7b15db29d6-kube-api-access-wnbq6\") pod \"distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") " pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.658623 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.658532    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 02 09:46:14.658623 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.658603    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492" podUID="ae2706aa-d41a-463a-acf6-79c6b2705f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 02 09:46:14.690016 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.689984    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:14.979062 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:14.978645    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"]
Nov 02 09:46:14.980896 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:46:14.980864    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda21c87af_b381_44b0_84f8_0f7b15db29d6.slice/crio-5b3bf6a3375ef24f0cb9790eb9cff7c9a90ac7fb93f2b8c631d9c02dfa855362 WatchSource:0}: Error finding container 5b3bf6a3375ef24f0cb9790eb9cff7c9a90ac7fb93f2b8c631d9c02dfa855362: Status 404 returned error can't find the container with id 5b3bf6a3375ef24f0cb9790eb9cff7c9a90ac7fb93f2b8c631d9c02dfa855362
Nov 02 09:46:15.771457 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:15.771414    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" event={"ID":"a21c87af-b381-44b0-84f8-0f7b15db29d6","Type":"ContainerStarted","Data":"831c2092bd42e41bb600b2addc3e6b92355667632c30cb6aa814f2600b422bf5"}
Nov 02 09:46:15.771945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:15.771475    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" event={"ID":"a21c87af-b381-44b0-84f8-0f7b15db29d6","Type":"ContainerStarted","Data":"5b3bf6a3375ef24f0cb9790eb9cff7c9a90ac7fb93f2b8c631d9c02dfa855362"}
Nov 02 09:46:15.796012 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:15.795952    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podStartSLOduration=1.7959330470000001 podStartE2EDuration="1.795933047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:15.793689408 +0000 UTC m=+656.626347488" watchObservedRunningTime="2026-11-02 09:46:15.795933047 +0000 UTC m=+656.628591128"
Nov 02 09:46:16.138844 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:16.138761    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:16.139209 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:16.139179    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:16.141136 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:16.141116    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 09:46:16.690313 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:16.690263    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:21.949725 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:21.949687    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"]
Nov 02 09:46:21.955562 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:21.955539    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"
Nov 02 09:46:21.960443 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:21.959456    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:46:21.960443 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:21.959581    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:46:21.962221 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:21.961659    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"]
Nov 02 09:46:22.004401 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:22.000498    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93664871-9gmj5\" (UID: \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"
Nov 02 09:46:22.004401 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:22.000556    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q5b6\" (UniqueName: \"kubernetes.io/projected/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-kube-api-access-2q5b6\") pod \"maas-api-revoke-sub-93664871-9gmj5\" (UID: \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"
Nov 02 09:46:22.004401 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:22.000618    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93664871-9gmj5\" (UID: \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"
Nov 02 09:46:22.101656 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:22.101626    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93664871-9gmj5\" (UID: \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"
Nov 02 09:46:22.101846 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:22.101661    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5b6\" (UniqueName: \"kubernetes.io/projected/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-kube-api-access-2q5b6\") pod \"maas-api-revoke-sub-93664871-9gmj5\" (UID: \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"
Nov 02 09:46:22.101846 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:22.101711    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93664871-9gmj5\" (UID: \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"
Nov 02 09:46:22.102425 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:22.102396    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93664871-9gmj5\" (UID: \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"
Nov 02 09:46:22.104172 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:22.104143    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93664871-9gmj5\" (UID: \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"
Nov 02 09:46:22.113931 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:22.113906    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q5b6\" (UniqueName: \"kubernetes.io/projected/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-kube-api-access-2q5b6\") pod \"maas-api-revoke-sub-93664871-9gmj5\" (UID: \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"
Nov 02 09:46:22.278943 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:22.278658    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"
Nov 02 09:46:22.317763 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:22.317726    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"]
Nov 02 09:46:22.646477 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:22.646450    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"]
Nov 02 09:46:22.648690 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:46:22.648655    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51871fc_4acd_40e7_b1fd_c012e2c5cebe.slice/crio-0b58cbba89b5cfbbc0e0aca5a505c159c0f0aa749678fb6020de161c2d94f4e7 WatchSource:0}: Error finding container 0b58cbba89b5cfbbc0e0aca5a505c159c0f0aa749678fb6020de161c2d94f4e7: Status 404 returned error can't find the container with id 0b58cbba89b5cfbbc0e0aca5a505c159c0f0aa749678fb6020de161c2d94f4e7
Nov 02 09:46:22.819004 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:22.818971    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5" event={"ID":"d51871fc-4acd-40e7-b1fd-c012e2c5cebe","Type":"ContainerStarted","Data":"0b58cbba89b5cfbbc0e0aca5a505c159c0f0aa749678fb6020de161c2d94f4e7"}
Nov 02 09:46:23.826724 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:23.826639    4614 generic.go:356] "Generic (PLEG): container finished" podID="d51871fc-4acd-40e7-b1fd-c012e2c5cebe" containerID="bbf86b81537eb490736edd2d96089282f9540ad59ffd02e3885e11c3ea4b0d57" exitCode=0
Nov 02 09:46:23.826724 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:23.826704    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5" event={"ID":"d51871fc-4acd-40e7-b1fd-c012e2c5cebe","Type":"ContainerDied","Data":"bbf86b81537eb490736edd2d96089282f9540ad59ffd02e3885e11c3ea4b0d57"}
Nov 02 09:46:24.690766 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:24.690737    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:24.703342 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:24.703320    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:24.844765 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:24.844738    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:46:24.994841 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:24.994816    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"
Nov 02 09:46:25.037521 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:25.037489    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-kube-api-access-2q5b6\" (UniqueName: \"kubernetes.io/projected/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-kube-api-access-2q5b6\") pod \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\" (UID: \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\") "
Nov 02 09:46:25.037687 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:25.037556    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-maas-api-openshift-ca\") pod \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\" (UID: \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\") "
Nov 02 09:46:25.037687 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:25.037582    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-maas-api-xks-ca\") pod \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\" (UID: \"d51871fc-4acd-40e7-b1fd-c012e2c5cebe\") "
Nov 02 09:46:25.037988 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:25.037958    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-maas-api-openshift-ca" pod "d51871fc-4acd-40e7-b1fd-c012e2c5cebe" (UID: "d51871fc-4acd-40e7-b1fd-c012e2c5cebe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:46:25.039826 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:25.039797    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-maas-api-xks-ca" pod "d51871fc-4acd-40e7-b1fd-c012e2c5cebe" (UID: "d51871fc-4acd-40e7-b1fd-c012e2c5cebe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:46:25.039958 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:25.039928    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-kube-api-access-2q5b6" pod "d51871fc-4acd-40e7-b1fd-c012e2c5cebe" (UID: "d51871fc-4acd-40e7-b1fd-c012e2c5cebe"). InnerVolumeSpecName "kube-api-access-2q5b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:25.138697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:25.138659    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q5b6\" (UniqueName: \"kubernetes.io/projected/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-kube-api-access-2q5b6\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:25.138697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:25.138693    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:25.138881 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:25.138708    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51871fc-4acd-40e7-b1fd-c012e2c5cebe-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:25.838376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:25.838326    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"
Nov 02 09:46:25.838565 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:25.838374    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5" event={"ID":"d51871fc-4acd-40e7-b1fd-c012e2c5cebe","Type":"ContainerDied","Data":"0b58cbba89b5cfbbc0e0aca5a505c159c0f0aa749678fb6020de161c2d94f4e7"}
Nov 02 09:46:25.838565 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:25.838423    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b58cbba89b5cfbbc0e0aca5a505c159c0f0aa749678fb6020de161c2d94f4e7"
Nov 02 09:46:27.325232 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:27.325196    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp" podUID="0293c383-90a8-40bd-b429-185b81da0831" containerName="payload-processing" containerID="cri-o://8f5d91e90b4da6ffec30e89dd8032e97d39dd1a9be0646a4cad99d50eefa708f" gracePeriod=25
Nov 02 09:46:28.791495 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:28.791451    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp" podUID="0293c383-90a8-40bd-b429-185b81da0831" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:46:37.498937 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.498903    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"]
Nov 02 09:46:37.499550 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.499364    4614 cpu_manager.go:414] "containerMap: removing container" podUID="d51871fc-4acd-40e7-b1fd-c012e2c5cebe" containerName="revoke-keys"
Nov 02 09:46:37.499550 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.499383    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d51871fc-4acd-40e7-b1fd-c012e2c5cebe" containerName="revoke-keys"
Nov 02 09:46:37.499550 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.499479    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="d51871fc-4acd-40e7-b1fd-c012e2c5cebe" containerName="revoke-keys"
Nov 02 09:46:37.503196 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.502749    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.523779 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.523750    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2e80fb-openshift-default-dockercfg-545dt\""
Nov 02 09:46:37.528634 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.528612    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"]
Nov 02 09:46:37.582840 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.582810    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.583012 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.582853    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fv4x\" (UniqueName: \"kubernetes.io/projected/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-kube-api-access-6fv4x\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.583012 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.582881    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.583114 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.583084    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.583167 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.583127    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.583253 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.583233    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.583313 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.583281    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.583424 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.583410    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.583490 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.583465    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.684672 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.684641    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.684843 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.684703    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.684843 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.684743    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.684843 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.684782    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fv4x\" (UniqueName: \"kubernetes.io/projected/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-kube-api-access-6fv4x\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.684843 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.684809    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.685091 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.684882    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.685151 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.685081    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.685207 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.685148    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.685207 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.685155    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.685367 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.685217    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.685656 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.685499    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.685656 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.685591    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.685905 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.685878    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.685990 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.685945    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.687311 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.687291    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.687537 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.687518    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.697299 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.697273    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.697773 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.697753    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fv4x\" (UniqueName: \"kubernetes.io/projected/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-kube-api-access-6fv4x\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:37.831856 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:37.831765    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:38.132867 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:38.132842    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"]
Nov 02 09:46:38.135930 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:46:38.135904    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c147ae_1d41_49a8_89f9_adf158a7ae5c.slice/crio-b0d600ee8380d861e2ba2c69c66f16d03dd430f3c584931eaae80ef6d2b2cc67 WatchSource:0}: Error finding container b0d600ee8380d861e2ba2c69c66f16d03dd430f3c584931eaae80ef6d2b2cc67: Status 404 returned error can't find the container with id b0d600ee8380d861e2ba2c69c66f16d03dd430f3c584931eaae80ef6d2b2cc67
Nov 02 09:46:38.139296 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:38.139261    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:46:38.139417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:38.139346    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:46:38.139488 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:38.139428    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:46:38.603374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:38.602646    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"]
Nov 02 09:46:38.603374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:38.602877    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf" podUID="065a3fb3-353d-4f77-868b-83e98ace1aca" containerName="istio-proxy" containerID="cri-o://470a4f0ca37814600c6156b541fa3e1ac8989344d552f29806cb086ef03045e8" gracePeriod=30
Nov 02 09:46:38.794965 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:38.794932    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp" podUID="0293c383-90a8-40bd-b429-185b81da0831" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:46:38.937543 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:38.937443    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" event={"ID":"d1c147ae-1d41-49a8-89f9-adf158a7ae5c","Type":"ContainerStarted","Data":"dc32472ab974d5194562c5f3038f313a8ddb8d3346c712c2a09d7fc254331db2"}
Nov 02 09:46:38.937806 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:38.937784    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" event={"ID":"d1c147ae-1d41-49a8-89f9-adf158a7ae5c","Type":"ContainerStarted","Data":"b0d600ee8380d861e2ba2c69c66f16d03dd430f3c584931eaae80ef6d2b2cc67"}
Nov 02 09:46:38.980081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:38.980013    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" podStartSLOduration=1.9799956010000002 podStartE2EDuration="1.979995601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:38.979340101 +0000 UTC m=+679.811998183" watchObservedRunningTime="2026-11-02 09:46:38.979995601 +0000 UTC m=+679.812653679"
Nov 02 09:46:39.656846 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:46:39.656808    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:46:39.845376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:39.836347    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:39.845376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:39.844844    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 02 09:46:39.845376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:39.844915    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 02 09:46:40.843375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:40.842723    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 02 09:46:40.843375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:40.842786    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 02 09:46:41.843459 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:41.843425    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 02 09:46:41.844044 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:41.844016    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 02 09:46:42.852373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:42.852305    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 02 09:46:42.852948 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:42.852916    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 02 09:46:43.833441 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:43.833406    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 02 09:46:43.833736 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:43.833703    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 02 09:46:43.994441 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:43.994409    4614 generic.go:356] "Generic (PLEG): container finished" podID="065a3fb3-353d-4f77-868b-83e98ace1aca" containerID="470a4f0ca37814600c6156b541fa3e1ac8989344d552f29806cb086ef03045e8" exitCode=0
Nov 02 09:46:43.994863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:43.994532    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf" event={"ID":"065a3fb3-353d-4f77-868b-83e98ace1aca","Type":"ContainerDied","Data":"470a4f0ca37814600c6156b541fa3e1ac8989344d552f29806cb086ef03045e8"}
Nov 02 09:46:43.994863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:43.994569    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf" event={"ID":"065a3fb3-353d-4f77-868b-83e98ace1aca","Type":"ContainerDied","Data":"96d43e0782023e61556e131242010a6e098d88e8bf0ba29eebce2af81701fbc2"}
Nov 02 09:46:43.994863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:43.994586    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96d43e0782023e61556e131242010a6e098d88e8bf0ba29eebce2af81701fbc2"
Nov 02 09:46:44.002030 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.002005    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:46:44.107015 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.106981    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-credential-socket\") pod \"065a3fb3-353d-4f77-868b-83e98ace1aca\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") "
Nov 02 09:46:44.107272 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.107084    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/065a3fb3-353d-4f77-868b-83e98ace1aca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065a3fb3-353d-4f77-868b-83e98ace1aca-istiod-ca-cert\") pod \"065a3fb3-353d-4f77-868b-83e98ace1aca\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") "
Nov 02 09:46:44.107272 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.107116    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-podinfo\") pod \"065a3fb3-353d-4f77-868b-83e98ace1aca\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") "
Nov 02 09:46:44.107272 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.107153    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-envoy\") pod \"065a3fb3-353d-4f77-868b-83e98ace1aca\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") "
Nov 02 09:46:44.107272 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.107192    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-workload-socket\") pod \"065a3fb3-353d-4f77-868b-83e98ace1aca\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") "
Nov 02 09:46:44.107272 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.107245    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-token\" (UniqueName: \"kubernetes.io/projected/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-token\") pod \"065a3fb3-353d-4f77-868b-83e98ace1aca\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") "
Nov 02 09:46:44.107606 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.107292    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-credential-socket" pod "065a3fb3-353d-4f77-868b-83e98ace1aca" (UID: "065a3fb3-353d-4f77-868b-83e98ace1aca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:44.107606 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.107308    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065a3fb3-353d-4f77-868b-83e98ace1aca-kube-api-access-7cpzj\" (UniqueName: \"kubernetes.io/projected/065a3fb3-353d-4f77-868b-83e98ace1aca-kube-api-access-7cpzj\") pod \"065a3fb3-353d-4f77-868b-83e98ace1aca\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") "
Nov 02 09:46:44.107606 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.107535    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065a3fb3-353d-4f77-868b-83e98ace1aca-istiod-ca-cert" pod "065a3fb3-353d-4f77-868b-83e98ace1aca" (UID: "065a3fb3-353d-4f77-868b-83e98ace1aca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:46:44.107606 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.107535    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-workload-socket" pod "065a3fb3-353d-4f77-868b-83e98ace1aca" (UID: "065a3fb3-353d-4f77-868b-83e98ace1aca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:44.107606 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.107591    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-workload-certs\") pod \"065a3fb3-353d-4f77-868b-83e98ace1aca\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") "
Nov 02 09:46:44.107864 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.107624    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-data\") pod \"065a3fb3-353d-4f77-868b-83e98ace1aca\" (UID: \"065a3fb3-353d-4f77-868b-83e98ace1aca\") "
Nov 02 09:46:44.107924 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.107869    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-workload-certs" pod "065a3fb3-353d-4f77-868b-83e98ace1aca" (UID: "065a3fb3-353d-4f77-868b-83e98ace1aca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:44.108051 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.108031    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-data" pod "065a3fb3-353d-4f77-868b-83e98ace1aca" (UID: "065a3fb3-353d-4f77-868b-83e98ace1aca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:44.108207 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.108189    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065a3fb3-353d-4f77-868b-83e98ace1aca-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.108277 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.108211    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.108277 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.108225    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.108277 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.108240    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.108277 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.108254    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.110532 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.110507    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065a3fb3-353d-4f77-868b-83e98ace1aca-kube-api-access-7cpzj" pod "065a3fb3-353d-4f77-868b-83e98ace1aca" (UID: "065a3fb3-353d-4f77-868b-83e98ace1aca"). InnerVolumeSpecName "kube-api-access-7cpzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:44.110649 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.110559    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-podinfo" pod "065a3fb3-353d-4f77-868b-83e98ace1aca" (UID: "065a3fb3-353d-4f77-868b-83e98ace1aca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:46:44.110649 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.110614    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-token" pod "065a3fb3-353d-4f77-868b-83e98ace1aca" (UID: "065a3fb3-353d-4f77-868b-83e98ace1aca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:44.110649 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.110572    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-envoy" pod "065a3fb3-353d-4f77-868b-83e98ace1aca" (UID: "065a3fb3-353d-4f77-868b-83e98ace1aca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:46:44.209513 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.209415    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.209513 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.209443    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.209513 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.209452    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065a3fb3-353d-4f77-868b-83e98ace1aca-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.209513 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.209462    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cpzj\" (UniqueName: \"kubernetes.io/projected/065a3fb3-353d-4f77-868b-83e98ace1aca-kube-api-access-7cpzj\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:44.837782 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.837329    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 02 09:46:44.837782 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:44.837410    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 02 09:46:45.001172 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:45.001137    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"
Nov 02 09:46:45.031657 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:45.031627    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"]
Nov 02 09:46:45.035749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:45.035720    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-000ffb-openshift-default-74665c677-tx8bf"]
Nov 02 09:46:45.316187 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:45.316152    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="065a3fb3-353d-4f77-868b-83e98ace1aca" path="/var/lib/kubelet/pods/065a3fb3-353d-4f77-868b-83e98ace1aca/volumes"
Nov 02 09:46:46.294317 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:46.294281    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:46.294835 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:46.294696    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:46.295690 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:46.295669    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:46:47.606227 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:47.606198    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"
Nov 02 09:46:47.657680 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:47.657646    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0293c383-90a8-40bd-b429-185b81da0831-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0293c383-90a8-40bd-b429-185b81da0831-plugins-config-volume\") pod \"0293c383-90a8-40bd-b429-185b81da0831\" (UID: \"0293c383-90a8-40bd-b429-185b81da0831\") "
Nov 02 09:46:47.657856 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:47.657720    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0293c383-90a8-40bd-b429-185b81da0831-kube-api-access-qtnlq\" (UniqueName: \"kubernetes.io/projected/0293c383-90a8-40bd-b429-185b81da0831-kube-api-access-qtnlq\") pod \"0293c383-90a8-40bd-b429-185b81da0831\" (UID: \"0293c383-90a8-40bd-b429-185b81da0831\") "
Nov 02 09:46:47.658049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:47.658026    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0293c383-90a8-40bd-b429-185b81da0831-plugins-config-volume" pod "0293c383-90a8-40bd-b429-185b81da0831" (UID: "0293c383-90a8-40bd-b429-185b81da0831"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:46:47.659776 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:47.659757    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0293c383-90a8-40bd-b429-185b81da0831-kube-api-access-qtnlq" pod "0293c383-90a8-40bd-b429-185b81da0831" (UID: "0293c383-90a8-40bd-b429-185b81da0831"). InnerVolumeSpecName "kube-api-access-qtnlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:47.758692 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:47.758653    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0293c383-90a8-40bd-b429-185b81da0831-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:47.758692 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:47.758683    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtnlq\" (UniqueName: \"kubernetes.io/projected/0293c383-90a8-40bd-b429-185b81da0831-kube-api-access-qtnlq\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:48.017823 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:48.017783    4614 generic.go:356] "Generic (PLEG): container finished" podID="0293c383-90a8-40bd-b429-185b81da0831" containerID="8f5d91e90b4da6ffec30e89dd8032e97d39dd1a9be0646a4cad99d50eefa708f" exitCode=0
Nov 02 09:46:48.018031 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:48.017834    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"
Nov 02 09:46:48.018031 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:48.017869    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp" event={"ID":"0293c383-90a8-40bd-b429-185b81da0831","Type":"ContainerDied","Data":"8f5d91e90b4da6ffec30e89dd8032e97d39dd1a9be0646a4cad99d50eefa708f"}
Nov 02 09:46:48.018031 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:48.017931    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp" event={"ID":"0293c383-90a8-40bd-b429-185b81da0831","Type":"ContainerDied","Data":"05d875fbec6a0dd44e93e31ec319ee23ca2916a5ecc961e6306b8e3eff3147ff"}
Nov 02 09:46:48.018031 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:48.017960    4614 scope.go:122] "RemoveContainer" containerID="8f5d91e90b4da6ffec30e89dd8032e97d39dd1a9be0646a4cad99d50eefa708f"
Nov 02 09:46:48.027634 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:48.027618    4614 scope.go:122] "RemoveContainer" containerID="8f5d91e90b4da6ffec30e89dd8032e97d39dd1a9be0646a4cad99d50eefa708f"
Nov 02 09:46:48.027876 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:46:48.027861    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5d91e90b4da6ffec30e89dd8032e97d39dd1a9be0646a4cad99d50eefa708f\": container with ID starting with 8f5d91e90b4da6ffec30e89dd8032e97d39dd1a9be0646a4cad99d50eefa708f not found: ID does not exist" containerID="8f5d91e90b4da6ffec30e89dd8032e97d39dd1a9be0646a4cad99d50eefa708f"
Nov 02 09:46:48.027933 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:48.027882    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5d91e90b4da6ffec30e89dd8032e97d39dd1a9be0646a4cad99d50eefa708f"} err="failed to get container status \"8f5d91e90b4da6ffec30e89dd8032e97d39dd1a9be0646a4cad99d50eefa708f\": rpc error: code = NotFound desc = could not find container \"8f5d91e90b4da6ffec30e89dd8032e97d39dd1a9be0646a4cad99d50eefa708f\": container with ID starting with 8f5d91e90b4da6ffec30e89dd8032e97d39dd1a9be0646a4cad99d50eefa708f not found: ID does not exist"
Nov 02 09:46:48.041013 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:48.040991    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"]
Nov 02 09:46:48.048854 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:48.048835    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-000ffb-69745656c9-lmgkp"]
Nov 02 09:46:49.314833 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:49.314800    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0293c383-90a8-40bd-b429-185b81da0831" path="/var/lib/kubelet/pods/0293c383-90a8-40bd-b429-185b81da0831/volumes"
Nov 02 09:46:50.461436 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.461391    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"]
Nov 02 09:46:50.461911 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.461892    4614 cpu_manager.go:414] "containerMap: removing container" podUID="065a3fb3-353d-4f77-868b-83e98ace1aca" containerName="istio-proxy"
Nov 02 09:46:50.461973 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.461913    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="065a3fb3-353d-4f77-868b-83e98ace1aca" containerName="istio-proxy"
Nov 02 09:46:50.461973 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.461948    4614 cpu_manager.go:414] "containerMap: removing container" podUID="0293c383-90a8-40bd-b429-185b81da0831" containerName="payload-processing"
Nov 02 09:46:50.461973 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.461957    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0293c383-90a8-40bd-b429-185b81da0831" containerName="payload-processing"
Nov 02 09:46:50.462099 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.462085    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="065a3fb3-353d-4f77-868b-83e98ace1aca" containerName="istio-proxy"
Nov 02 09:46:50.462167 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.462103    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="0293c383-90a8-40bd-b429-185b81da0831" containerName="payload-processing"
Nov 02 09:46:50.466463 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.466438    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"
Nov 02 09:46:50.474650 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.474408    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:46:50.474650 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.474590    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"]
Nov 02 09:46:50.474829 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.474708    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:46:50.492237 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.492206    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a0587e-2931-40a4-a73b-be4a43a40030-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0188acf8-wkhq9\" (UID: \"94a0587e-2931-40a4-a73b-be4a43a40030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"
Nov 02 09:46:50.492396 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.492244    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgns\" (UniqueName: \"kubernetes.io/projected/94a0587e-2931-40a4-a73b-be4a43a40030-kube-api-access-nkgns\") pod \"maas-api-revoke-sub-0188acf8-wkhq9\" (UID: \"94a0587e-2931-40a4-a73b-be4a43a40030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"
Nov 02 09:46:50.492396 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.492370    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a0587e-2931-40a4-a73b-be4a43a40030-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0188acf8-wkhq9\" (UID: \"94a0587e-2931-40a4-a73b-be4a43a40030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"
Nov 02 09:46:50.595818 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.595776    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a0587e-2931-40a4-a73b-be4a43a40030-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0188acf8-wkhq9\" (UID: \"94a0587e-2931-40a4-a73b-be4a43a40030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"
Nov 02 09:46:50.596020 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.595905    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a0587e-2931-40a4-a73b-be4a43a40030-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0188acf8-wkhq9\" (UID: \"94a0587e-2931-40a4-a73b-be4a43a40030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"
Nov 02 09:46:50.596020 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.595939    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgns\" (UniqueName: \"kubernetes.io/projected/94a0587e-2931-40a4-a73b-be4a43a40030-kube-api-access-nkgns\") pod \"maas-api-revoke-sub-0188acf8-wkhq9\" (UID: \"94a0587e-2931-40a4-a73b-be4a43a40030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"
Nov 02 09:46:50.597304 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.597279    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a0587e-2931-40a4-a73b-be4a43a40030-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0188acf8-wkhq9\" (UID: \"94a0587e-2931-40a4-a73b-be4a43a40030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"
Nov 02 09:46:50.604695 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.604664    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a0587e-2931-40a4-a73b-be4a43a40030-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0188acf8-wkhq9\" (UID: \"94a0587e-2931-40a4-a73b-be4a43a40030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"
Nov 02 09:46:50.611402 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.611371    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgns\" (UniqueName: \"kubernetes.io/projected/94a0587e-2931-40a4-a73b-be4a43a40030-kube-api-access-nkgns\") pod \"maas-api-revoke-sub-0188acf8-wkhq9\" (UID: \"94a0587e-2931-40a4-a73b-be4a43a40030\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"
Nov 02 09:46:50.781530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:50.781503    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"
Nov 02 09:46:51.060951 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:51.060918    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"]
Nov 02 09:46:51.063611 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:46:51.063585    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a0587e_2931_40a4_a73b_be4a43a40030.slice/crio-75ebaa6e52366d885e4fbb2fa697fee3063184a2052b9696a0d71c5d67d51501 WatchSource:0}: Error finding container 75ebaa6e52366d885e4fbb2fa697fee3063184a2052b9696a0d71c5d67d51501: Status 404 returned error can't find the container with id 75ebaa6e52366d885e4fbb2fa697fee3063184a2052b9696a0d71c5d67d51501
Nov 02 09:46:51.843191 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:46:51.843154    4614 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a0587e_2931_40a4_a73b_be4a43a40030.slice/crio-2ef7d7570cfba1592e85f1fde35f438e5ab6823f7bf1f94a427456c0ccf8f1bf.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 09:46:52.041997 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.041959    4614 generic.go:356] "Generic (PLEG): container finished" podID="94a0587e-2931-40a4-a73b-be4a43a40030" containerID="2ef7d7570cfba1592e85f1fde35f438e5ab6823f7bf1f94a427456c0ccf8f1bf" exitCode=0
Nov 02 09:46:52.042181 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.042025    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9" event={"ID":"94a0587e-2931-40a4-a73b-be4a43a40030","Type":"ContainerDied","Data":"2ef7d7570cfba1592e85f1fde35f438e5ab6823f7bf1f94a427456c0ccf8f1bf"}
Nov 02 09:46:52.042181 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.042102    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9" event={"ID":"94a0587e-2931-40a4-a73b-be4a43a40030","Type":"ContainerStarted","Data":"75ebaa6e52366d885e4fbb2fa697fee3063184a2052b9696a0d71c5d67d51501"}
Nov 02 09:46:52.450177 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.450142    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"]
Nov 02 09:46:52.455978 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.455955    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"]
Nov 02 09:46:52.456126 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.456103    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.458280 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.458262    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.466316 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.466250    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"]
Nov 02 09:46:52.468106 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.468084    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"]
Nov 02 09:46:52.547920 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.547887    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.548081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.547932    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz4k9\" (UniqueName: \"kubernetes.io/projected/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-kube-api-access-tz4k9\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.548081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.547960    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.548186 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548078    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.548186 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548108    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.548186 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548135    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.548306 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548198    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.548306 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548215    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.548306 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548233    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rrm\" (UniqueName: \"kubernetes.io/projected/041c3aa0-45b9-42f1-a86c-0cd1e4404593-kube-api-access-w2rrm\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.548306 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548260    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.548545 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548297    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.548545 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548527    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.548650 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548578    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.548650 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548614    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.548754 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548653    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.548754 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548688    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.548754 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548730    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.548906 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.548755    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.649778 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.649745    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.649969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.649799    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.649969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.649831    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.649969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.649863    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.650143 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650004    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.650143 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650076    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.650143 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650108    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.650296 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650156    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.650296 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650206    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4k9\" (UniqueName: \"kubernetes.io/projected/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-kube-api-access-tz4k9\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.650296 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650255    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.650296 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650287    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.650528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650290    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.650528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650310    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.650528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650316    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.650528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650293    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.650528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650395    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.650528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650435    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.650528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650463    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.650528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650497    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rrm\" (UniqueName: \"kubernetes.io/projected/041c3aa0-45b9-42f1-a86c-0cd1e4404593-kube-api-access-w2rrm\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.650934 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650544    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.650934 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650571    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.650934 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650607    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.650934 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650627    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.650934 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650612    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.650934 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.650856    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.651241 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.651134    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.651316 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.651294    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.651566 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.651543    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.652627 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.652602    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.652751 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.652608    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.652927 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.652907    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.652983 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.652960    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.665070 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.665012    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.665070 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.665022    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.665232 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.665173    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rrm\" (UniqueName: \"kubernetes.io/projected/041c3aa0-45b9-42f1-a86c-0cd1e4404593-kube-api-access-w2rrm\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.665232 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.665211    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4k9\" (UniqueName: \"kubernetes.io/projected/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-kube-api-access-tz4k9\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:52.775473 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.775445    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:52.783424 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:52.783398    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:53.080202 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.080115    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"]
Nov 02 09:46:53.083498 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.083476    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"]
Nov 02 09:46:53.086523 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:46:53.086495    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041c3aa0_45b9_42f1_a86c_0cd1e4404593.slice/crio-e7e1d3a99e72d222f846ea9ca6042dde0139934e8897ddcd95a6fad44ed2372f WatchSource:0}: Error finding container e7e1d3a99e72d222f846ea9ca6042dde0139934e8897ddcd95a6fad44ed2372f: Status 404 returned error can't find the container with id e7e1d3a99e72d222f846ea9ca6042dde0139934e8897ddcd95a6fad44ed2372f
Nov 02 09:46:53.090504 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.090465    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:46:53.090606 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.090550    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:46:53.090606 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.090591    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:46:53.092481 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.092445    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:46:53.092584 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.092548    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:46:53.092781 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.092604    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:46:53.342424 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.338565    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"
Nov 02 09:46:53.385517 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.384795    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a0587e-2931-40a4-a73b-be4a43a40030-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a0587e-2931-40a4-a73b-be4a43a40030-maas-api-openshift-ca\") pod \"94a0587e-2931-40a4-a73b-be4a43a40030\" (UID: \"94a0587e-2931-40a4-a73b-be4a43a40030\") "
Nov 02 09:46:53.385517 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.384992    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a0587e-2931-40a4-a73b-be4a43a40030-kube-api-access-nkgns\" (UniqueName: \"kubernetes.io/projected/94a0587e-2931-40a4-a73b-be4a43a40030-kube-api-access-nkgns\") pod \"94a0587e-2931-40a4-a73b-be4a43a40030\" (UID: \"94a0587e-2931-40a4-a73b-be4a43a40030\") "
Nov 02 09:46:53.385517 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.385065    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94a0587e-2931-40a4-a73b-be4a43a40030-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a0587e-2931-40a4-a73b-be4a43a40030-maas-api-xks-ca\") pod \"94a0587e-2931-40a4-a73b-be4a43a40030\" (UID: \"94a0587e-2931-40a4-a73b-be4a43a40030\") "
Nov 02 09:46:53.387170 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.387120    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a0587e-2931-40a4-a73b-be4a43a40030-maas-api-openshift-ca" pod "94a0587e-2931-40a4-a73b-be4a43a40030" (UID: "94a0587e-2931-40a4-a73b-be4a43a40030"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:46:53.399549 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.399492    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a0587e-2931-40a4-a73b-be4a43a40030-kube-api-access-nkgns" pod "94a0587e-2931-40a4-a73b-be4a43a40030" (UID: "94a0587e-2931-40a4-a73b-be4a43a40030"). InnerVolumeSpecName "kube-api-access-nkgns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:46:53.411020 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.410984    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a0587e-2931-40a4-a73b-be4a43a40030-maas-api-xks-ca" pod "94a0587e-2931-40a4-a73b-be4a43a40030" (UID: "94a0587e-2931-40a4-a73b-be4a43a40030"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:46:53.487067 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.486973    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a0587e-2931-40a4-a73b-be4a43a40030-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:53.487067 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.487020    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkgns\" (UniqueName: \"kubernetes.io/projected/94a0587e-2931-40a4-a73b-be4a43a40030-kube-api-access-nkgns\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:53.487067 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:53.487037    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a0587e-2931-40a4-a73b-be4a43a40030-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:46:54.126379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.124905    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" event={"ID":"041c3aa0-45b9-42f1-a86c-0cd1e4404593","Type":"ContainerStarted","Data":"c27db4bf0858e319748be554a041cef4a225f4879d5dc35f01f26e6754f0bd38"}
Nov 02 09:46:54.126379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.125019    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" event={"ID":"041c3aa0-45b9-42f1-a86c-0cd1e4404593","Type":"ContainerStarted","Data":"e7e1d3a99e72d222f846ea9ca6042dde0139934e8897ddcd95a6fad44ed2372f"}
Nov 02 09:46:54.145384 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.143706    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9" event={"ID":"94a0587e-2931-40a4-a73b-be4a43a40030","Type":"ContainerDied","Data":"75ebaa6e52366d885e4fbb2fa697fee3063184a2052b9696a0d71c5d67d51501"}
Nov 02 09:46:54.145384 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.143773    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75ebaa6e52366d885e4fbb2fa697fee3063184a2052b9696a0d71c5d67d51501"
Nov 02 09:46:54.145384 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.143886    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"
Nov 02 09:46:54.149087 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.149056    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" event={"ID":"58f3dc8a-8a94-48b8-b09f-41cf77f122f5","Type":"ContainerStarted","Data":"7ddb4b3487498c3ca60b67429bc4fc1b60f9427141f4022f4c4cc3701778086c"}
Nov 02 09:46:54.149312 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.149296    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" event={"ID":"58f3dc8a-8a94-48b8-b09f-41cf77f122f5","Type":"ContainerStarted","Data":"4eef4fa69e8b96601d2e097de9753a403754818b6380e9a1ea301fd6278e672b"}
Nov 02 09:46:54.221074 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.219305    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" podStartSLOduration=2.219289008 podStartE2EDuration="2.219289008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:54.16376273 +0000 UTC m=+694.996420811" watchObservedRunningTime="2026-11-02 09:46:54.219289008 +0000 UTC m=+695.051947088"
Nov 02 09:46:54.221214 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.221172    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" podStartSLOduration=2.221165953 podStartE2EDuration="2.221165953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:54.218841122 +0000 UTC m=+695.051499203" watchObservedRunningTime="2026-11-02 09:46:54.221165953 +0000 UTC m=+695.053824032"
Nov 02 09:46:54.780377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.779413    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:46:54.780377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.779552    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 09:46:54.780377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.779595    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" podUID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 09:46:54.789376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.786808    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:46:54.794376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.790469    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 09:46:54.794376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:54.790519    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 09:46:55.779160 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:55.778754    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 09:46:55.779160 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:55.778817    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" podUID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 09:46:55.784631 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:55.784603    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 09:46:55.784777 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:55.784645    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 09:46:56.777843 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:56.777795    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 09:46:56.778086 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:56.777860    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" podUID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 09:46:56.784124 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:56.784097    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 09:46:56.784495 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:56.784141    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 09:46:57.792954 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:57.792919    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 09:46:57.793562 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:57.793528    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" podUID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 09:46:57.795625 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:57.795473    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 09:46:57.795625 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:57.795512    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 09:46:57.800969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:57.800943    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"]
Nov 02 09:46:57.801495 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:57.801474    4614 cpu_manager.go:414] "containerMap: removing container" podUID="94a0587e-2931-40a4-a73b-be4a43a40030" containerName="revoke-keys"
Nov 02 09:46:57.801592 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:57.801497    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a0587e-2931-40a4-a73b-be4a43a40030" containerName="revoke-keys"
Nov 02 09:46:57.801685 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:57.801669    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a0587e-2931-40a4-a73b-be4a43a40030" containerName="revoke-keys"
Nov 02 09:46:57.806106 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:57.806088    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"
Nov 02 09:46:57.815174 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:57.815148    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"]
Nov 02 09:46:57.819035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:57.818543    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8743a5-dockercfg-zbwgb\""
Nov 02 09:46:57.819035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:57.818546    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8743a5\""
Nov 02 09:46:57.893307 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:57.893272    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b9d779-f66b-4283-8eef-d1f80751fef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr\" (UID: \"29b9d779-f66b-4283-8eef-d1f80751fef1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"
Nov 02 09:46:57.893821 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:57.893792    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc65k\" (UniqueName: \"kubernetes.io/projected/29b9d779-f66b-4283-8eef-d1f80751fef1-kube-api-access-zc65k\") pod \"payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr\" (UID: \"29b9d779-f66b-4283-8eef-d1f80751fef1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"
Nov 02 09:46:58.057098 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:58.057000    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc65k\" (UniqueName: \"kubernetes.io/projected/29b9d779-f66b-4283-8eef-d1f80751fef1-kube-api-access-zc65k\") pod \"payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr\" (UID: \"29b9d779-f66b-4283-8eef-d1f80751fef1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"
Nov 02 09:46:58.058856 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:58.058831    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b9d779-f66b-4283-8eef-d1f80751fef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr\" (UID: \"29b9d779-f66b-4283-8eef-d1f80751fef1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"
Nov 02 09:46:58.072289 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:58.071812    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b9d779-f66b-4283-8eef-d1f80751fef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr\" (UID: \"29b9d779-f66b-4283-8eef-d1f80751fef1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"
Nov 02 09:46:58.087445 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:58.087405    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc65k\" (UniqueName: \"kubernetes.io/projected/29b9d779-f66b-4283-8eef-d1f80751fef1-kube-api-access-zc65k\") pod \"payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr\" (UID: \"29b9d779-f66b-4283-8eef-d1f80751fef1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"
Nov 02 09:46:58.125436 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:58.125337    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"
Nov 02 09:46:58.520417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:58.520385    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"]
Nov 02 09:46:58.524557 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:46:58.524531    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b9d779_f66b_4283_8eef_d1f80751fef1.slice/crio-c4e65b6edafb3ed34fd6a65a28d96ba2e604cbb3abf4a3305fe462d6f2349f4f WatchSource:0}: Error finding container c4e65b6edafb3ed34fd6a65a28d96ba2e604cbb3abf4a3305fe462d6f2349f4f: Status 404 returned error can't find the container with id c4e65b6edafb3ed34fd6a65a28d96ba2e604cbb3abf4a3305fe462d6f2349f4f
Nov 02 09:46:58.786425 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:58.780748    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 09:46:58.786425 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:58.780822    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" podUID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 09:46:58.795375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:58.792769    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 09:46:58.795375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:58.792826    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 09:46:59.217432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:59.217314    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr" event={"ID":"29b9d779-f66b-4283-8eef-d1f80751fef1","Type":"ContainerStarted","Data":"cc912e937b666b975f1a8c1db05a019241f44868b89f376d9d029b32e0c4f26e"}
Nov 02 09:46:59.217432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:59.217415    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr" event={"ID":"29b9d779-f66b-4283-8eef-d1f80751fef1","Type":"ContainerStarted","Data":"c4e65b6edafb3ed34fd6a65a28d96ba2e604cbb3abf4a3305fe462d6f2349f4f"}
Nov 02 09:46:59.217649 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:59.217447    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"
Nov 02 09:46:59.238746 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:59.238677    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr" podStartSLOduration=2.238658576 podStartE2EDuration="2.238658576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:46:59.234687395 +0000 UTC m=+700.067345477" watchObservedRunningTime="2026-11-02 09:46:59.238658576 +0000 UTC m=+700.071316656"
Nov 02 09:46:59.776444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:59.776394    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 09:46:59.776834 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:59.776485    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" podUID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 09:46:59.784702 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:59.784664    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 09:46:59.784823 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:46:59.784736    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 09:47:00.437348 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.437312    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"]
Nov 02 09:47:00.441161 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.441143    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"
Nov 02 09:47:00.446523 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.446501    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:47:00.447699 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.447676    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:47:00.450415 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.450391    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"]
Nov 02 09:47:00.534108 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.534076    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a614f25-277c-4485-9c5e-93630dd5bb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4d2766-jzkmh\" (UID: \"4a614f25-277c-4485-9c5e-93630dd5bb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"
Nov 02 09:47:00.534278 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.534129    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a614f25-277c-4485-9c5e-93630dd5bb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4d2766-jzkmh\" (UID: \"4a614f25-277c-4485-9c5e-93630dd5bb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"
Nov 02 09:47:00.534278 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.534253    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tksxw\" (UniqueName: \"kubernetes.io/projected/4a614f25-277c-4485-9c5e-93630dd5bb2f-kube-api-access-tksxw\") pod \"maas-api-revoke-sub-5e4d2766-jzkmh\" (UID: \"4a614f25-277c-4485-9c5e-93630dd5bb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"
Nov 02 09:47:00.635720 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.635678    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a614f25-277c-4485-9c5e-93630dd5bb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4d2766-jzkmh\" (UID: \"4a614f25-277c-4485-9c5e-93630dd5bb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"
Nov 02 09:47:00.635912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.635755    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a614f25-277c-4485-9c5e-93630dd5bb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4d2766-jzkmh\" (UID: \"4a614f25-277c-4485-9c5e-93630dd5bb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"
Nov 02 09:47:00.635912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.635798    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tksxw\" (UniqueName: \"kubernetes.io/projected/4a614f25-277c-4485-9c5e-93630dd5bb2f-kube-api-access-tksxw\") pod \"maas-api-revoke-sub-5e4d2766-jzkmh\" (UID: \"4a614f25-277c-4485-9c5e-93630dd5bb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"
Nov 02 09:47:00.636514 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.636487    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a614f25-277c-4485-9c5e-93630dd5bb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4d2766-jzkmh\" (UID: \"4a614f25-277c-4485-9c5e-93630dd5bb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"
Nov 02 09:47:00.638303 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.638270    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a614f25-277c-4485-9c5e-93630dd5bb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4d2766-jzkmh\" (UID: \"4a614f25-277c-4485-9c5e-93630dd5bb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"
Nov 02 09:47:00.648798 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.648774    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tksxw\" (UniqueName: \"kubernetes.io/projected/4a614f25-277c-4485-9c5e-93630dd5bb2f-kube-api-access-tksxw\") pod \"maas-api-revoke-sub-5e4d2766-jzkmh\" (UID: \"4a614f25-277c-4485-9c5e-93630dd5bb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"
Nov 02 09:47:00.764376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.763543    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"
Nov 02 09:47:00.779784 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.779746    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 09:47:00.779952 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.779827    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" podUID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 09:47:00.791040 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.787679    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 09:47:00.791040 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:00.787765    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 09:47:01.083424 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:01.082925    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"]
Nov 02 09:47:01.235939 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:01.235895    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh" event={"ID":"4a614f25-277c-4485-9c5e-93630dd5bb2f","Type":"ContainerStarted","Data":"e5f1505deaa366398e2f0b7a5e80dc5edfd6113b975efe5d216b3c0a52f74277"}
Nov 02 09:47:01.789603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:01.789567    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:47:01.790055 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:01.789881    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:47:01.790716 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:01.790695    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 09:47:01.818098 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:01.818068    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:47:01.819329 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:01.819231    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:47:01.825036 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:01.823487    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 09:47:02.317867 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:02.317827    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh" event={"ID":"4a614f25-277c-4485-9c5e-93630dd5bb2f","Type":"ContainerStarted","Data":"2915783aead53101faa0ba561b33b6d33445dd493921291a99c227349b3cff3e"}
Nov 02 09:47:02.351118 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:02.351053    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh" podStartSLOduration=2.351036979 podStartE2EDuration="2.351036979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:02.347260716 +0000 UTC m=+703.179918793" watchObservedRunningTime="2026-11-02 09:47:02.351036979 +0000 UTC m=+703.183695059"
Nov 02 09:47:03.323759 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:03.323726    4614 generic.go:356] "Generic (PLEG): container finished" podID="4a614f25-277c-4485-9c5e-93630dd5bb2f" containerID="2915783aead53101faa0ba561b33b6d33445dd493921291a99c227349b3cff3e" exitCode=0
Nov 02 09:47:03.324186 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:03.323812    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh" event={"ID":"4a614f25-277c-4485-9c5e-93630dd5bb2f","Type":"ContainerDied","Data":"2915783aead53101faa0ba561b33b6d33445dd493921291a99c227349b3cff3e"}
Nov 02 09:47:04.466752 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:04.466725    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"
Nov 02 09:47:04.628620 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:04.628542    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a614f25-277c-4485-9c5e-93630dd5bb2f-kube-api-access-tksxw\" (UniqueName: \"kubernetes.io/projected/4a614f25-277c-4485-9c5e-93630dd5bb2f-kube-api-access-tksxw\") pod \"4a614f25-277c-4485-9c5e-93630dd5bb2f\" (UID: \"4a614f25-277c-4485-9c5e-93630dd5bb2f\") "
Nov 02 09:47:04.628770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:04.628667    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a614f25-277c-4485-9c5e-93630dd5bb2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a614f25-277c-4485-9c5e-93630dd5bb2f-maas-api-openshift-ca\") pod \"4a614f25-277c-4485-9c5e-93630dd5bb2f\" (UID: \"4a614f25-277c-4485-9c5e-93630dd5bb2f\") "
Nov 02 09:47:04.628770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:04.628704    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a614f25-277c-4485-9c5e-93630dd5bb2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a614f25-277c-4485-9c5e-93630dd5bb2f-maas-api-xks-ca\") pod \"4a614f25-277c-4485-9c5e-93630dd5bb2f\" (UID: \"4a614f25-277c-4485-9c5e-93630dd5bb2f\") "
Nov 02 09:47:04.629031 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:04.629001    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a614f25-277c-4485-9c5e-93630dd5bb2f-maas-api-openshift-ca" pod "4a614f25-277c-4485-9c5e-93630dd5bb2f" (UID: "4a614f25-277c-4485-9c5e-93630dd5bb2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:47:04.630728 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:04.630700    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a614f25-277c-4485-9c5e-93630dd5bb2f-kube-api-access-tksxw" pod "4a614f25-277c-4485-9c5e-93630dd5bb2f" (UID: "4a614f25-277c-4485-9c5e-93630dd5bb2f"). InnerVolumeSpecName "kube-api-access-tksxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:04.630831 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:04.630800    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a614f25-277c-4485-9c5e-93630dd5bb2f-maas-api-xks-ca" pod "4a614f25-277c-4485-9c5e-93630dd5bb2f" (UID: "4a614f25-277c-4485-9c5e-93630dd5bb2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:47:04.729810 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:04.729761    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a614f25-277c-4485-9c5e-93630dd5bb2f-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:04.729810 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:04.729798    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a614f25-277c-4485-9c5e-93630dd5bb2f-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:04.729810 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:04.729813    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tksxw\" (UniqueName: \"kubernetes.io/projected/4a614f25-277c-4485-9c5e-93630dd5bb2f-kube-api-access-tksxw\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:05.339387 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:05.339333    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh" event={"ID":"4a614f25-277c-4485-9c5e-93630dd5bb2f","Type":"ContainerDied","Data":"e5f1505deaa366398e2f0b7a5e80dc5edfd6113b975efe5d216b3c0a52f74277"}
Nov 02 09:47:05.339534 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:05.339405    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"
Nov 02 09:47:05.339534 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:05.339406    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5f1505deaa366398e2f0b7a5e80dc5edfd6113b975efe5d216b3c0a52f74277"
Nov 02 09:47:06.429861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.429818    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"]
Nov 02 09:47:06.430318 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.430301    4614 cpu_manager.go:414] "containerMap: removing container" podUID="4a614f25-277c-4485-9c5e-93630dd5bb2f" containerName="revoke-keys"
Nov 02 09:47:06.430391 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.430320    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a614f25-277c-4485-9c5e-93630dd5bb2f" containerName="revoke-keys"
Nov 02 09:47:06.430468 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.430457    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a614f25-277c-4485-9c5e-93630dd5bb2f" containerName="revoke-keys"
Nov 02 09:47:06.433900 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.433871    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"
Nov 02 09:47:06.442442 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.442416    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"]
Nov 02 09:47:06.503073 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.503044    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:47:06.503228 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.503085    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:47:06.548745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.548714    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx74b\" (UniqueName: \"kubernetes.io/projected/916f8630-c2f0-431f-92d2-0a677bb0921a-kube-api-access-xx74b\") pod \"maas-api-revoke-sub-8b22726a-sw7kx\" (UID: \"916f8630-c2f0-431f-92d2-0a677bb0921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"
Nov 02 09:47:06.548922 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.548784    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916f8630-c2f0-431f-92d2-0a677bb0921a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b22726a-sw7kx\" (UID: \"916f8630-c2f0-431f-92d2-0a677bb0921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"
Nov 02 09:47:06.548922 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.548899    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916f8630-c2f0-431f-92d2-0a677bb0921a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b22726a-sw7kx\" (UID: \"916f8630-c2f0-431f-92d2-0a677bb0921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"
Nov 02 09:47:06.650629 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.650580    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916f8630-c2f0-431f-92d2-0a677bb0921a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b22726a-sw7kx\" (UID: \"916f8630-c2f0-431f-92d2-0a677bb0921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"
Nov 02 09:47:06.650783 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.650719    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916f8630-c2f0-431f-92d2-0a677bb0921a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b22726a-sw7kx\" (UID: \"916f8630-c2f0-431f-92d2-0a677bb0921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"
Nov 02 09:47:06.650844 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.650808    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx74b\" (UniqueName: \"kubernetes.io/projected/916f8630-c2f0-431f-92d2-0a677bb0921a-kube-api-access-xx74b\") pod \"maas-api-revoke-sub-8b22726a-sw7kx\" (UID: \"916f8630-c2f0-431f-92d2-0a677bb0921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"
Nov 02 09:47:06.651569 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.651542    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916f8630-c2f0-431f-92d2-0a677bb0921a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b22726a-sw7kx\" (UID: \"916f8630-c2f0-431f-92d2-0a677bb0921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"
Nov 02 09:47:06.653324 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.653298    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916f8630-c2f0-431f-92d2-0a677bb0921a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b22726a-sw7kx\" (UID: \"916f8630-c2f0-431f-92d2-0a677bb0921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"
Nov 02 09:47:06.663363 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.663325    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx74b\" (UniqueName: \"kubernetes.io/projected/916f8630-c2f0-431f-92d2-0a677bb0921a-kube-api-access-xx74b\") pod \"maas-api-revoke-sub-8b22726a-sw7kx\" (UID: \"916f8630-c2f0-431f-92d2-0a677bb0921a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"
Nov 02 09:47:06.749753 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:06.749718    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"
Nov 02 09:47:07.058658 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:07.058627    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"]
Nov 02 09:47:07.060247 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:47:07.060218    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916f8630_c2f0_431f_92d2_0a677bb0921a.slice/crio-6d4f8b0cf762c641385e00e7a5c8cc93ba744958ce36473451484d4ccfd9f221 WatchSource:0}: Error finding container 6d4f8b0cf762c641385e00e7a5c8cc93ba744958ce36473451484d4ccfd9f221: Status 404 returned error can't find the container with id 6d4f8b0cf762c641385e00e7a5c8cc93ba744958ce36473451484d4ccfd9f221
Nov 02 09:47:07.352089 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:07.352058    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx" event={"ID":"916f8630-c2f0-431f-92d2-0a677bb0921a","Type":"ContainerStarted","Data":"bb7a9ec02c01b1c597d6ff2f9899f21ace320eef982a9d2632a2bf75f0efab89"}
Nov 02 09:47:07.352226 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:07.352109    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx" event={"ID":"916f8630-c2f0-431f-92d2-0a677bb0921a","Type":"ContainerStarted","Data":"6d4f8b0cf762c641385e00e7a5c8cc93ba744958ce36473451484d4ccfd9f221"}
Nov 02 09:47:07.371010 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:07.370963    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx" podStartSLOduration=1.370948759 podStartE2EDuration="1.370948759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:07.370032773 +0000 UTC m=+708.202690977" watchObservedRunningTime="2026-11-02 09:47:07.370948759 +0000 UTC m=+708.203606838"
Nov 02 09:47:08.358864 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:08.358834    4614 generic.go:356] "Generic (PLEG): container finished" podID="916f8630-c2f0-431f-92d2-0a677bb0921a" containerID="bb7a9ec02c01b1c597d6ff2f9899f21ace320eef982a9d2632a2bf75f0efab89" exitCode=0
Nov 02 09:47:08.359231 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:08.358893    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx" event={"ID":"916f8630-c2f0-431f-92d2-0a677bb0921a","Type":"ContainerDied","Data":"bb7a9ec02c01b1c597d6ff2f9899f21ace320eef982a9d2632a2bf75f0efab89"}
Nov 02 09:47:09.514019 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:09.513994    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"
Nov 02 09:47:09.676820 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:47:09.676735    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:47:09.688871 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:09.688832    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916f8630-c2f0-431f-92d2-0a677bb0921a-kube-api-access-xx74b\" (UniqueName: \"kubernetes.io/projected/916f8630-c2f0-431f-92d2-0a677bb0921a-kube-api-access-xx74b\") pod \"916f8630-c2f0-431f-92d2-0a677bb0921a\" (UID: \"916f8630-c2f0-431f-92d2-0a677bb0921a\") "
Nov 02 09:47:09.689141 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:09.689121    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/916f8630-c2f0-431f-92d2-0a677bb0921a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916f8630-c2f0-431f-92d2-0a677bb0921a-maas-api-xks-ca\") pod \"916f8630-c2f0-431f-92d2-0a677bb0921a\" (UID: \"916f8630-c2f0-431f-92d2-0a677bb0921a\") "
Nov 02 09:47:09.689301 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:09.689284    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916f8630-c2f0-431f-92d2-0a677bb0921a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916f8630-c2f0-431f-92d2-0a677bb0921a-maas-api-openshift-ca\") pod \"916f8630-c2f0-431f-92d2-0a677bb0921a\" (UID: \"916f8630-c2f0-431f-92d2-0a677bb0921a\") "
Nov 02 09:47:09.690284 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:09.690257    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916f8630-c2f0-431f-92d2-0a677bb0921a-maas-api-openshift-ca" pod "916f8630-c2f0-431f-92d2-0a677bb0921a" (UID: "916f8630-c2f0-431f-92d2-0a677bb0921a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:47:09.693378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:09.693325    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916f8630-c2f0-431f-92d2-0a677bb0921a-kube-api-access-xx74b" pod "916f8630-c2f0-431f-92d2-0a677bb0921a" (UID: "916f8630-c2f0-431f-92d2-0a677bb0921a"). InnerVolumeSpecName "kube-api-access-xx74b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:09.697500 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:09.697474    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916f8630-c2f0-431f-92d2-0a677bb0921a-maas-api-xks-ca" pod "916f8630-c2f0-431f-92d2-0a677bb0921a" (UID: "916f8630-c2f0-431f-92d2-0a677bb0921a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:47:09.790787 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:09.790750    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx74b\" (UniqueName: \"kubernetes.io/projected/916f8630-c2f0-431f-92d2-0a677bb0921a-kube-api-access-xx74b\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:09.790787 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:09.790786    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916f8630-c2f0-431f-92d2-0a677bb0921a-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:09.791041 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:09.790813    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916f8630-c2f0-431f-92d2-0a677bb0921a-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:10.228916 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:10.228874    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"
Nov 02 09:47:10.372804 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:10.372769    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx" event={"ID":"916f8630-c2f0-431f-92d2-0a677bb0921a","Type":"ContainerDied","Data":"6d4f8b0cf762c641385e00e7a5c8cc93ba744958ce36473451484d4ccfd9f221"}
Nov 02 09:47:10.373006 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:10.372820    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d4f8b0cf762c641385e00e7a5c8cc93ba744958ce36473451484d4ccfd9f221"
Nov 02 09:47:10.373006 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:10.372873    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"
Nov 02 09:47:10.906717 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:10.906674    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"]
Nov 02 09:47:10.930375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:10.907344    4614 cpu_manager.go:414] "containerMap: removing container" podUID="916f8630-c2f0-431f-92d2-0a677bb0921a" containerName="revoke-keys"
Nov 02 09:47:10.930375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:10.908675    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916f8630-c2f0-431f-92d2-0a677bb0921a" containerName="revoke-keys"
Nov 02 09:47:10.930375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:10.908937    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="916f8630-c2f0-431f-92d2-0a677bb0921a" containerName="revoke-keys"
Nov 02 09:47:10.930375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:10.913169    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:10.936862 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:10.931514    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-629129-serving-cert\""
Nov 02 09:47:10.936862 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:10.935059    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"]
Nov 02 09:47:10.937737 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:10.937591    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5thw\""
Nov 02 09:47:10.937737 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:10.937703    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 09:47:11.031944 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:11.031906    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3ea645-ba8e-4ebb-8332-1617c13436be-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:11.031944 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:11.031954    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxxwk\" (UniqueName: \"kubernetes.io/projected/7e3ea645-ba8e-4ebb-8332-1617c13436be-kube-api-access-cxxwk\") pod \"maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:11.032190 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:11.031988    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-maas-api-tls\") pod \"maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:11.032190 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:11.032158    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-metrics-certs\") pod \"maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:11.138376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:11.135940    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3ea645-ba8e-4ebb-8332-1617c13436be-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:11.138376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:11.135982    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxxwk\" (UniqueName: \"kubernetes.io/projected/7e3ea645-ba8e-4ebb-8332-1617c13436be-kube-api-access-cxxwk\") pod \"maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:11.138376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:11.136009    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-maas-api-tls\") pod \"maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:11.138376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:11.136055    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-metrics-certs\") pod \"maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:11.138376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:11.137770    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3ea645-ba8e-4ebb-8332-1617c13436be-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:11.138376 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:47:11.137862    4614 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-serving-cert: secret "maas-api-e2e-extmod-a-629129-serving-cert" not found
Nov 02 09:47:11.138376 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:47:11.137914    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-maas-api-tls podName:7e3ea645-ba8e-4ebb-8332-1617c13436be nodeName:}" failed. No retries permitted until 2026-11-02 09:47:11.637895994 +0000 UTC m=+712.470554053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-maas-api-tls") pod "maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm" (UID: "7e3ea645-ba8e-4ebb-8332-1617c13436be") : secret "maas-api-e2e-extmod-a-629129-serving-cert" not found
Nov 02 09:47:11.163676 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:11.157729    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxxwk\" (UniqueName: \"kubernetes.io/projected/7e3ea645-ba8e-4ebb-8332-1617c13436be-kube-api-access-cxxwk\") pod \"maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:11.169055 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:11.168813    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-metrics-certs\") pod \"maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:11.680376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:11.668089    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-maas-api-tls\") pod \"maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:11.714377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:11.707944    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-maas-api-tls\") pod \"maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:11.862428 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:11.861709    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:12.472377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:12.469818    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"]
Nov 02 09:47:12.494492 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:47:12.494455    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3ea645_ba8e_4ebb_8332_1617c13436be.slice/crio-bb0ecced87a1404f5becc5acb6c6710f1e8265e72fd1f40550cd11c64ad26fa3 WatchSource:0}: Error finding container bb0ecced87a1404f5becc5acb6c6710f1e8265e72fd1f40550cd11c64ad26fa3: Status 404 returned error can't find the container with id bb0ecced87a1404f5becc5acb6c6710f1e8265e72fd1f40550cd11c64ad26fa3
Nov 02 09:47:12.498939 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:12.498916    4614 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 09:47:13.442450 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:13.442410    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm" event={"ID":"7e3ea645-ba8e-4ebb-8332-1617c13436be","Type":"ContainerStarted","Data":"b9ef58d191fc7d2b4074388b731b5d97995e784c318c368ac429cbe4d694554c"}
Nov 02 09:47:13.442663 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:13.442470    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm" event={"ID":"7e3ea645-ba8e-4ebb-8332-1617c13436be","Type":"ContainerStarted","Data":"bb0ecced87a1404f5becc5acb6c6710f1e8265e72fd1f40550cd11c64ad26fa3"}
Nov 02 09:47:13.442663 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:13.442633    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:13.465770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:13.465724    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm" podStartSLOduration=3.011291073 podStartE2EDuration="3.465710444s" totalImagesPullingTime="454.419371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:13.463494815 +0000 UTC m=+714.296152896" watchObservedRunningTime="2026-11-02 09:47:13.465710444 +0000 UTC m=+714.298368524"
Nov 02 09:47:15.468284 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.468245    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"]
Nov 02 09:47:15.472282 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.472255    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"
Nov 02 09:47:15.478508 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.478484    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:47:15.479013 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.478985    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:47:15.481492 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.481468    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"]
Nov 02 09:47:15.562312 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.562280    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a36b689-w692q\" (UID: \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"
Nov 02 09:47:15.562515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.562337    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtrk5\" (UniqueName: \"kubernetes.io/projected/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-kube-api-access-qtrk5\") pod \"maas-api-revoke-sub-7a36b689-w692q\" (UID: \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"
Nov 02 09:47:15.562570 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.562522    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a36b689-w692q\" (UID: \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"
Nov 02 09:47:15.663466 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.663435    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a36b689-w692q\" (UID: \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"
Nov 02 09:47:15.663658 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.663499    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a36b689-w692q\" (UID: \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"
Nov 02 09:47:15.663658 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.663537    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtrk5\" (UniqueName: \"kubernetes.io/projected/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-kube-api-access-qtrk5\") pod \"maas-api-revoke-sub-7a36b689-w692q\" (UID: \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"
Nov 02 09:47:15.664224 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.664203    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a36b689-w692q\" (UID: \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"
Nov 02 09:47:15.665898 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.665875    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a36b689-w692q\" (UID: \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"
Nov 02 09:47:15.675577 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.675554    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtrk5\" (UniqueName: \"kubernetes.io/projected/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-kube-api-access-qtrk5\") pod \"maas-api-revoke-sub-7a36b689-w692q\" (UID: \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"
Nov 02 09:47:15.786732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:15.786658    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"
Nov 02 09:47:16.063617 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:16.063587    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"]
Nov 02 09:47:16.065505 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:47:16.065477    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7c07ca_855c_4498_bd0f_7dc808ab78cd.slice/crio-61fc2e93397164e28e04dd6886bc507eb21af5d0a754ce2473751cadf26f03a0 WatchSource:0}: Error finding container 61fc2e93397164e28e04dd6886bc507eb21af5d0a754ce2473751cadf26f03a0: Status 404 returned error can't find the container with id 61fc2e93397164e28e04dd6886bc507eb21af5d0a754ce2473751cadf26f03a0
Nov 02 09:47:16.462112 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:16.462080    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q" event={"ID":"1b7c07ca-855c-4498-bd0f-7dc808ab78cd","Type":"ContainerStarted","Data":"29d9f91790c567f3b7eb6bd48012ef940ee4d8136e410c0ead11e55820bbf8dc"}
Nov 02 09:47:16.462242 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:16.462133    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q" event={"ID":"1b7c07ca-855c-4498-bd0f-7dc808ab78cd","Type":"ContainerStarted","Data":"61fc2e93397164e28e04dd6886bc507eb21af5d0a754ce2473751cadf26f03a0"}
Nov 02 09:47:16.481678 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:16.481638    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q" podStartSLOduration=1.4816244969999999 podStartE2EDuration="1.481624497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:16.479505664 +0000 UTC m=+717.312163745" watchObservedRunningTime="2026-11-02 09:47:16.481624497 +0000 UTC m=+717.314282556"
Nov 02 09:47:17.469338 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:17.469307    4614 generic.go:356] "Generic (PLEG): container finished" podID="1b7c07ca-855c-4498-bd0f-7dc808ab78cd" containerID="29d9f91790c567f3b7eb6bd48012ef940ee4d8136e410c0ead11e55820bbf8dc" exitCode=0
Nov 02 09:47:17.469522 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:17.469412    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q" event={"ID":"1b7c07ca-855c-4498-bd0f-7dc808ab78cd","Type":"ContainerDied","Data":"29d9f91790c567f3b7eb6bd48012ef940ee4d8136e410c0ead11e55820bbf8dc"}
Nov 02 09:47:18.621851 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:18.621823    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"
Nov 02 09:47:18.714296 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:18.714265    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-kube-api-access-qtrk5\" (UniqueName: \"kubernetes.io/projected/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-kube-api-access-qtrk5\") pod \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\" (UID: \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\") "
Nov 02 09:47:18.714499 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:18.714389    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-maas-api-openshift-ca\") pod \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\" (UID: \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\") "
Nov 02 09:47:18.714499 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:18.714414    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-maas-api-xks-ca\") pod \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\" (UID: \"1b7c07ca-855c-4498-bd0f-7dc808ab78cd\") "
Nov 02 09:47:18.714776 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:18.714745    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-maas-api-openshift-ca" pod "1b7c07ca-855c-4498-bd0f-7dc808ab78cd" (UID: "1b7c07ca-855c-4498-bd0f-7dc808ab78cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:47:18.716374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:18.716337    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-maas-api-xks-ca" pod "1b7c07ca-855c-4498-bd0f-7dc808ab78cd" (UID: "1b7c07ca-855c-4498-bd0f-7dc808ab78cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:47:18.716462 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:18.716375    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-kube-api-access-qtrk5" pod "1b7c07ca-855c-4498-bd0f-7dc808ab78cd" (UID: "1b7c07ca-855c-4498-bd0f-7dc808ab78cd"). InnerVolumeSpecName "kube-api-access-qtrk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:18.815990 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:18.815917    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:18.815990 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:18.815945    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:18.815990 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:18.815956    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtrk5\" (UniqueName: \"kubernetes.io/projected/1b7c07ca-855c-4498-bd0f-7dc808ab78cd-kube-api-access-qtrk5\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:19.454641 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:19.454613    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 09:47:19.482788 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:19.482763    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"
Nov 02 09:47:19.482931 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:19.482805    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q" event={"ID":"1b7c07ca-855c-4498-bd0f-7dc808ab78cd","Type":"ContainerDied","Data":"61fc2e93397164e28e04dd6886bc507eb21af5d0a754ce2473751cadf26f03a0"}
Nov 02 09:47:19.482931 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:19.482855    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61fc2e93397164e28e04dd6886bc507eb21af5d0a754ce2473751cadf26f03a0"
Nov 02 09:47:26.236468 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.236424    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"]
Nov 02 09:47:26.237065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.237042    4614 cpu_manager.go:414] "containerMap: removing container" podUID="1b7c07ca-855c-4498-bd0f-7dc808ab78cd" containerName="revoke-keys"
Nov 02 09:47:26.237152 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.237069    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7c07ca-855c-4498-bd0f-7dc808ab78cd" containerName="revoke-keys"
Nov 02 09:47:26.237222 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.237211    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7c07ca-855c-4498-bd0f-7dc808ab78cd" containerName="revoke-keys"
Nov 02 09:47:26.242392 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.242369    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.250201 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.250180    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-9e7156-openshift-default-dockercfg-nlp5g\""
Nov 02 09:47:26.255395 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.255100    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"]
Nov 02 09:47:26.399994 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.399952    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.400176 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.400018    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94175a6-949c-4995-b110-b9749c2920b6-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.400176 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.400126    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94175a6-949c-4995-b110-b9749c2920b6-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.400176 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.400166    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.400317 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.400182    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94175a6-949c-4995-b110-b9749c2920b6-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.400317 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.400210    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.400317 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.400308    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.400478 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.400333    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2n82\" (UniqueName: \"kubernetes.io/projected/c94175a6-949c-4995-b110-b9749c2920b6-kube-api-access-t2n82\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.400478 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.400464    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.501439 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.501332    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.501439 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.501415    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94175a6-949c-4995-b110-b9749c2920b6-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.501673 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.501466    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94175a6-949c-4995-b110-b9749c2920b6-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.501673 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.501498    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.501673 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.501518    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94175a6-949c-4995-b110-b9749c2920b6-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.501673 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.501564    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.501673 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.501634    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.501673 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.501651    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2n82\" (UniqueName: \"kubernetes.io/projected/c94175a6-949c-4995-b110-b9749c2920b6-kube-api-access-t2n82\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.501980 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.501674    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.502032 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.501984    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.502087 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.502030    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.502087 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.502074    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.502260 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.502237    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.502365 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.502334    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94175a6-949c-4995-b110-b9749c2920b6-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.504686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.504656    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.504686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.504665    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94175a6-949c-4995-b110-b9749c2920b6-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.514123 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.514096    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94175a6-949c-4995-b110-b9749c2920b6-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.514538 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.514515    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2n82\" (UniqueName: \"kubernetes.io/projected/c94175a6-949c-4995-b110-b9749c2920b6-kube-api-access-t2n82\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.557858 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.557821    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:26.846219 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.846193    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"]
Nov 02 09:47:26.847974 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:47:26.847944    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94175a6_949c_4995_b110_b9749c2920b6.slice/crio-7bc3da263472cb4679154333a5d94eaa22b204b19a65cb290d48173d7ade6ec1 WatchSource:0}: Error finding container 7bc3da263472cb4679154333a5d94eaa22b204b19a65cb290d48173d7ade6ec1: Status 404 returned error can't find the container with id 7bc3da263472cb4679154333a5d94eaa22b204b19a65cb290d48173d7ade6ec1
Nov 02 09:47:26.850346 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.850310    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:26.850495 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.850401    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:26.850495 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:26.850466    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:27.597074 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:27.597042    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv" event={"ID":"c94175a6-949c-4995-b110-b9749c2920b6","Type":"ContainerStarted","Data":"348d499c0112424af3f34d6800c4b29c68b4c09177e08de96541bf3d97ef60e1"}
Nov 02 09:47:27.597492 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:27.597091    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv" event={"ID":"c94175a6-949c-4995-b110-b9749c2920b6","Type":"ContainerStarted","Data":"7bc3da263472cb4679154333a5d94eaa22b204b19a65cb290d48173d7ade6ec1"}
Nov 02 09:47:27.624642 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:27.624598    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv" podStartSLOduration=1.624584185 podStartE2EDuration="1.624584185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:27.620871395 +0000 UTC m=+728.453529475" watchObservedRunningTime="2026-11-02 09:47:27.624584185 +0000 UTC m=+728.457242264"
Nov 02 09:47:28.558712 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:28.558680    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:28.560725 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:28.560695    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 09:47:28.560878 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:28.560752    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv" podUID="c94175a6-949c-4995-b110-b9749c2920b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 09:47:29.074146 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:29.074112    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"]
Nov 02 09:47:29.559219 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:29.559181    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 09:47:29.559426 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:29.559247    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv" podUID="c94175a6-949c-4995-b110-b9749c2920b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 09:47:30.564029 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:30.562754    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 09:47:30.564029 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:30.562814    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv" podUID="c94175a6-949c-4995-b110-b9749c2920b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 09:47:31.558419 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:31.558385    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 09:47:31.558608 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:31.558440    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv" podUID="c94175a6-949c-4995-b110-b9749c2920b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 09:47:32.559141 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.559109    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 09:47:32.559571 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.559163    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv" podUID="c94175a6-949c-4995-b110-b9749c2920b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 09:47:32.785737 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.783673    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"]
Nov 02 09:47:32.788916 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.788890    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:32.799027 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.799003    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-204fba-openshift-default-dockercfg-xvjxq\""
Nov 02 09:47:32.807970 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.807942    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"]
Nov 02 09:47:32.916322 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.916230    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:32.916634 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.916608    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:32.916776 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.916759    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:32.916905 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.916885    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:32.916997 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.916941    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:32.917069 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.917054    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:32.917127 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.917113    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:32.917179 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.917152    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7gvv\" (UniqueName: \"kubernetes.io/projected/a0a1b0ce-e12b-4227-90be-f17a736ab20d-kube-api-access-g7gvv\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:32.917232 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:32.917194    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.018991 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.018949    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7gvv\" (UniqueName: \"kubernetes.io/projected/a0a1b0ce-e12b-4227-90be-f17a736ab20d-kube-api-access-g7gvv\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.019332 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.019307    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.019494 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.019478    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.019599 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.019586    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.019699 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.019684    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.019821 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.019806    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.019945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.019932    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.020139 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.020124    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.020265 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.020252    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.028376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.020783    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.029201 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.029173    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.030011 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.029984    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.044376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.030920    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.047207 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.046881    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.088908 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.088829    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.100394 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.095696    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.105385 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.103538    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7gvv\" (UniqueName: \"kubernetes.io/projected/a0a1b0ce-e12b-4227-90be-f17a736ab20d-kube-api-access-g7gvv\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.118237 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.118175    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.405938 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.405903    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:33.809510 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.808187    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"]
Nov 02 09:47:33.821365 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.820124    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:33.821365 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.820212    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:33.821365 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:33.820265    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:34.610497 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:34.610455    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:47:34.611099 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:34.611063    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv" podUID="c94175a6-949c-4995-b110-b9749c2920b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:34.736996 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:34.736958    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm" event={"ID":"a0a1b0ce-e12b-4227-90be-f17a736ab20d","Type":"ContainerStarted","Data":"f9e7be1164127f4bdff0792e229b1baf8b546dc62c84e4ed84eb70b583652e26"}
Nov 02 09:47:34.737323 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:34.737294    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm" event={"ID":"a0a1b0ce-e12b-4227-90be-f17a736ab20d","Type":"ContainerStarted","Data":"1dc40291cc5d6692dd3327c00870398e510895cfce4ab0ae666678a60095b11e"}
Nov 02 09:47:34.814372 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:34.813747    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm" podStartSLOduration=2.813727091 podStartE2EDuration="2.813727091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:34.81169891 +0000 UTC m=+735.644356991" watchObservedRunningTime="2026-11-02 09:47:34.813727091 +0000 UTC m=+735.646385172"
Nov 02 09:47:35.022374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:35.021704    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:35.033624 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:35.023050    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:35.034623 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:35.034598    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:47:35.409422 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:35.408872    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:35.418710 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:35.418681    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:35.623999 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:35.621439    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"]
Nov 02 09:47:35.840378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:35.798343    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:35.840378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:35.808791    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:47:37.661629 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.661584    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"]
Nov 02 09:47:37.667139 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.667102    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.683636 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.683608    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"]
Nov 02 09:47:37.693278 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.693245    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.693581 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.693299    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.693581 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.693430    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.693581 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.693468    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.693581 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.693528    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.693982 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.693578    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.693982 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.693701    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.693982 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.693785    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.693982 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.693846    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h94g\" (UniqueName: \"kubernetes.io/projected/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-kube-api-access-9h94g\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.698577 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.698551    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"]
Nov 02 09:47:37.703100 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.703079    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.725839 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.725811    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"]
Nov 02 09:47:37.794594 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.794554    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.794745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.794638    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.794745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.794661    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.794745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.794685    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839be18b-58e9-404a-8dfa-3a85e612879d-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.794745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.794713    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.794900 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.794862    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j6f7\" (UniqueName: \"kubernetes.io/projected/839be18b-58e9-404a-8dfa-3a85e612879d-kube-api-access-5j6f7\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.794931 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.794899    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.794969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.794934    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h94g\" (UniqueName: \"kubernetes.io/projected/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-kube-api-access-9h94g\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.794969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.794960    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839be18b-58e9-404a-8dfa-3a85e612879d-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.795051 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.795001    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.795051 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.795023    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.795116 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.795074    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.795157 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.795113    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.795193 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.795155    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.795193 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.795185    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.795289 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.795212    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.795349 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.795298    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839be18b-58e9-404a-8dfa-3a85e612879d-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.795445 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.795392    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.795504 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.795460    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.795618 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.795595    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.795686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.795648    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.795738 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.795705    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.796183 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.796161    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.797643 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.797620    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.797736 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.797625    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.808230 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.808183    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.808464 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.808443    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h94g\" (UniqueName: \"kubernetes.io/projected/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-kube-api-access-9h94g\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:37.897063 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.897022    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.897063 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.897063    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.897294 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.897115    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.897294 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.897141    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839be18b-58e9-404a-8dfa-3a85e612879d-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.897294 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.897168    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j6f7\" (UniqueName: \"kubernetes.io/projected/839be18b-58e9-404a-8dfa-3a85e612879d-kube-api-access-5j6f7\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.897294 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.897196    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.897294 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.897241    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839be18b-58e9-404a-8dfa-3a85e612879d-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.897570 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.897325    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.897570 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.897383    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839be18b-58e9-404a-8dfa-3a85e612879d-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.897684 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.897586    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.897684 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.897623    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.897963 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.897939    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.898056 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.898032    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839be18b-58e9-404a-8dfa-3a85e612879d-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.898151 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.898136    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.899427 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.899406    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.899714 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.899697    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839be18b-58e9-404a-8dfa-3a85e612879d-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.911812 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.911752    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839be18b-58e9-404a-8dfa-3a85e612879d-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.911812 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.911791    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j6f7\" (UniqueName: \"kubernetes.io/projected/839be18b-58e9-404a-8dfa-3a85e612879d-kube-api-access-5j6f7\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:37.983416 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:37.983386    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:38.021998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.021965    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:38.273860 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.273832    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"]
Nov 02 09:47:38.275462 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:47:38.275338    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858c3cf7_f3f4_41cf_a30e_d8dc86ed0ca6.slice/crio-1ead096d1b31bd93e966ab9c7f7bdc42492744b76e00aadfaf820cd72831bebc WatchSource:0}: Error finding container 1ead096d1b31bd93e966ab9c7f7bdc42492744b76e00aadfaf820cd72831bebc: Status 404 returned error can't find the container with id 1ead096d1b31bd93e966ab9c7f7bdc42492744b76e00aadfaf820cd72831bebc
Nov 02 09:47:38.278409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.278378    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:38.278509 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.278445    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:38.278509 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.278481    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:38.319063 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.319033    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"]
Nov 02 09:47:38.325454 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:47:38.324494    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839be18b_58e9_404a_8dfa_3a85e612879d.slice/crio-bde8237f641e96e3074febd6c0d6c0f0a98c600da379d0c7d38196a9074c1047 WatchSource:0}: Error finding container bde8237f641e96e3074febd6c0d6c0f0a98c600da379d0c7d38196a9074c1047: Status 404 returned error can't find the container with id bde8237f641e96e3074febd6c0d6c0f0a98c600da379d0c7d38196a9074c1047
Nov 02 09:47:38.327439 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.327400    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:38.327521 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.327490    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:38.327560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.327533    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:38.487148 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.487111    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"]
Nov 02 09:47:38.491772 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.491512    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"
Nov 02 09:47:38.506344 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.503502    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-204fba\""
Nov 02 09:47:38.508040 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.508015    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"]
Nov 02 09:47:38.521231 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.521207    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"]
Nov 02 09:47:38.527896 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.527837    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"
Nov 02 09:47:38.533281 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.533243    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-204fba-dockercfg-8qdrd\""
Nov 02 09:47:38.540896 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.540869    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"]
Nov 02 09:47:38.609649 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.609594    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf91158-4d01-4d00-8b19-84b4e0e48064-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7\" (UID: \"ebf91158-4d01-4d00-8b19-84b4e0e48064\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"
Nov 02 09:47:38.609825 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.609687    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgwbq\" (UniqueName: \"kubernetes.io/projected/3f19d824-07ba-46e9-b281-edd1a2dba5fd-kube-api-access-sgwbq\") pod \"payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr\" (UID: \"3f19d824-07ba-46e9-b281-edd1a2dba5fd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"
Nov 02 09:47:38.609825 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.609806    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f19d824-07ba-46e9-b281-edd1a2dba5fd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr\" (UID: \"3f19d824-07ba-46e9-b281-edd1a2dba5fd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"
Nov 02 09:47:38.609906 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.609855    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-745pv\" (UniqueName: \"kubernetes.io/projected/ebf91158-4d01-4d00-8b19-84b4e0e48064-kube-api-access-745pv\") pod \"payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7\" (UID: \"ebf91158-4d01-4d00-8b19-84b4e0e48064\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"
Nov 02 09:47:38.714080 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.714036    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf91158-4d01-4d00-8b19-84b4e0e48064-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7\" (UID: \"ebf91158-4d01-4d00-8b19-84b4e0e48064\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"
Nov 02 09:47:38.715591 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.715569    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgwbq\" (UniqueName: \"kubernetes.io/projected/3f19d824-07ba-46e9-b281-edd1a2dba5fd-kube-api-access-sgwbq\") pod \"payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr\" (UID: \"3f19d824-07ba-46e9-b281-edd1a2dba5fd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"
Nov 02 09:47:38.716173 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.716150    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f19d824-07ba-46e9-b281-edd1a2dba5fd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr\" (UID: \"3f19d824-07ba-46e9-b281-edd1a2dba5fd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"
Nov 02 09:47:38.730552 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.730526    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-745pv\" (UniqueName: \"kubernetes.io/projected/ebf91158-4d01-4d00-8b19-84b4e0e48064-kube-api-access-745pv\") pod \"payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7\" (UID: \"ebf91158-4d01-4d00-8b19-84b4e0e48064\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"
Nov 02 09:47:38.744375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.715464    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf91158-4d01-4d00-8b19-84b4e0e48064-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7\" (UID: \"ebf91158-4d01-4d00-8b19-84b4e0e48064\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"
Nov 02 09:47:38.744375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.720314    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f19d824-07ba-46e9-b281-edd1a2dba5fd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr\" (UID: \"3f19d824-07ba-46e9-b281-edd1a2dba5fd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"
Nov 02 09:47:38.745799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.745774    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgwbq\" (UniqueName: \"kubernetes.io/projected/3f19d824-07ba-46e9-b281-edd1a2dba5fd-kube-api-access-sgwbq\") pod \"payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr\" (UID: \"3f19d824-07ba-46e9-b281-edd1a2dba5fd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"
Nov 02 09:47:38.749065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.749041    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-745pv\" (UniqueName: \"kubernetes.io/projected/ebf91158-4d01-4d00-8b19-84b4e0e48064-kube-api-access-745pv\") pod \"payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7\" (UID: \"ebf91158-4d01-4d00-8b19-84b4e0e48064\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"
Nov 02 09:47:38.819487 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.818971    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"
Nov 02 09:47:38.843210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.842924    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" event={"ID":"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6","Type":"ContainerStarted","Data":"efddfbc2ab0f1282b4203098b1532f46efc8e4922ada95d46d61cc4612c9ff59"}
Nov 02 09:47:38.843210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.842997    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" event={"ID":"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6","Type":"ContainerStarted","Data":"1ead096d1b31bd93e966ab9c7f7bdc42492744b76e00aadfaf820cd72831bebc"}
Nov 02 09:47:38.852083 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.850836    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" event={"ID":"839be18b-58e9-404a-8dfa-3a85e612879d","Type":"ContainerStarted","Data":"da6a20935ef9e943dd667f209bf77aa43da36b80490e18c05d30d3c469260fdf"}
Nov 02 09:47:38.852083 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.850894    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" event={"ID":"839be18b-58e9-404a-8dfa-3a85e612879d","Type":"ContainerStarted","Data":"bde8237f641e96e3074febd6c0d6c0f0a98c600da379d0c7d38196a9074c1047"}
Nov 02 09:47:38.857770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.857747    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"
Nov 02 09:47:38.903787 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.903735    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" podStartSLOduration=1.903720737 podStartE2EDuration="1.903720737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:38.899938294 +0000 UTC m=+739.732596374" watchObservedRunningTime="2026-11-02 09:47:38.903720737 +0000 UTC m=+739.736378816"
Nov 02 09:47:38.939893 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.939817    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" podStartSLOduration=1.9397935450000001 podStartE2EDuration="1.939793545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:38.931483907 +0000 UTC m=+739.764141989" watchObservedRunningTime="2026-11-02 09:47:38.939793545 +0000 UTC m=+739.772451626"
Nov 02 09:47:38.983829 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:38.983739    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:39.024455 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.024015    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:39.024455 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.024139    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 09:47:39.024455 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.024178    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 09:47:39.196541 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.195820    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"]
Nov 02 09:47:39.206827 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.206801    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"]
Nov 02 09:47:39.210154 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:47:39.210127    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf91158_4d01_4d00_8b19_84b4e0e48064.slice/crio-a81ba73ab8d683ff183ac3d8b8c589eca1a65851210ec31835f396dec3095608 WatchSource:0}: Error finding container a81ba73ab8d683ff183ac3d8b8c589eca1a65851210ec31835f396dec3095608: Status 404 returned error can't find the container with id a81ba73ab8d683ff183ac3d8b8c589eca1a65851210ec31835f396dec3095608
Nov 02 09:47:39.706925 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:47:39.706884    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:47:39.919409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.917516    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr" event={"ID":"3f19d824-07ba-46e9-b281-edd1a2dba5fd","Type":"ContainerStarted","Data":"66c4c1bfc2cad55b6c9f3525c6dec536b0214a463c09f356993c131543d9855a"}
Nov 02 09:47:39.919409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.917568    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr" event={"ID":"3f19d824-07ba-46e9-b281-edd1a2dba5fd","Type":"ContainerStarted","Data":"16cbe7b72a823511e1d614090f1bc3f427fef3e26fb30a8573fd1ed197cfed35"}
Nov 02 09:47:39.919409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.918270    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"
Nov 02 09:47:39.920398 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.920336    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7" event={"ID":"ebf91158-4d01-4d00-8b19-84b4e0e48064","Type":"ContainerStarted","Data":"141bea154f7f4c908daed0373ad7c2c87760e7c09d754257d522e48565b5b300"}
Nov 02 09:47:39.920545 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.920433    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"
Nov 02 09:47:39.920633 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.920582    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7" event={"ID":"ebf91158-4d01-4d00-8b19-84b4e0e48064","Type":"ContainerStarted","Data":"a81ba73ab8d683ff183ac3d8b8c589eca1a65851210ec31835f396dec3095608"}
Nov 02 09:47:39.942378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.939207    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr" podStartSLOduration=1.939185985 podStartE2EDuration="1.939185985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:39.935745927 +0000 UTC m=+740.768404008" watchObservedRunningTime="2026-11-02 09:47:39.939185985 +0000 UTC m=+740.771844066"
Nov 02 09:47:39.964432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.962023    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7" podStartSLOduration=1.962001262 podStartE2EDuration="1.962001262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:39.958831589 +0000 UTC m=+740.791489671" watchObservedRunningTime="2026-11-02 09:47:39.962001262 +0000 UTC m=+740.794659342"
Nov 02 09:47:39.986242 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.986201    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 02 09:47:39.986442 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:39.986257    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" podUID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 02 09:47:40.022984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:40.022956    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 09:47:40.023118 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:40.023001    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 09:47:40.987540 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:40.987507    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 02 09:47:40.988344 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:40.988220    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" podUID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 02 09:47:41.023084 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:41.023058    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 09:47:41.023225 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:41.023098    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 09:47:41.991000 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:41.990966    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 02 09:47:41.991610 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:41.991582    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" podUID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 02 09:47:42.022496 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:42.022467    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 09:47:42.022731 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:42.022710    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 09:47:42.983689 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:42.983654    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 02 09:47:42.983866 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:42.983705    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" podUID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 02 09:47:43.032377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.028776    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 09:47:43.032377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.028845    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 09:47:43.873375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.872889    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"]
Nov 02 09:47:43.890767 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.890734    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:43.901725 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.901052    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-301be8-openshift-default-dockercfg-fxzjg\""
Nov 02 09:47:43.909600 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.905706    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"]
Nov 02 09:47:43.916440 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.916406    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-workload-certs\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:43.916585 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.916540    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-data\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:43.916585 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.916578    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvbbr\" (UniqueName: \"kubernetes.io/projected/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-kube-api-access-dvbbr\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:43.916698 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.916662    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-credential-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:43.916752 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.916722    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-workload-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:43.916816 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.916797    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-podinfo\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:43.916872 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.916845    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-envoy\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:43.916926 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.916880    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istiod-ca-cert\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:43.916980 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.916965    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-token\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:43.984913 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.984878    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 02 09:47:43.985193 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:43.985162    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" podUID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 02 09:47:44.018101 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.018059    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-data\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.018880 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.018858    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvbbr\" (UniqueName: \"kubernetes.io/projected/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-kube-api-access-dvbbr\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.019442 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.018779    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-data\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.019609 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.019393    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-credential-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.019725 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.019703    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-credential-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.019844 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.019772    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-workload-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.019932 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.019922    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-podinfo\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.020001 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.019992    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-envoy\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.020063 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.020042    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-workload-socket\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.020123 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.020112    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istiod-ca-cert\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.020205 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.020197    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-token\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.020304 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.020295    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-workload-certs\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.020666 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.020647    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-workload-certs\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.021187 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.021167    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istiod-ca-cert\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.022315 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.022294    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 09:47:44.022410 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.022344    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 09:47:44.022570 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.022549    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-envoy\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.023271 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.023252    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-podinfo\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.078238 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.078207    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvbbr\" (UniqueName: \"kubernetes.io/projected/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-kube-api-access-dvbbr\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.096759 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.096714    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r" podUID="57cde931-be21-4545-bd3c-df85b7170196" containerName="main" containerID="cri-o://a76dfafb09a5ca42c060a2a5f115b02dbccf1d7b82b47876f7686c68d9c7b8d6" gracePeriod=45
Nov 02 09:47:44.098836 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.098790    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-token\") pod \"e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") " pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.213849 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.213755    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:44.998411 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.987178    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 02 09:47:44.998411 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:44.987253    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" podUID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 02 09:47:45.006863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.006566    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:47:45.050378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.033007    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"]
Nov 02 09:47:45.053150 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.053124    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 09:47:45.053363 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.053320    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 09:47:45.101573 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.094102    4614 generic.go:356] "Generic (PLEG): container finished" podID="57cde931-be21-4545-bd3c-df85b7170196" containerID="a76dfafb09a5ca42c060a2a5f115b02dbccf1d7b82b47876f7686c68d9c7b8d6" exitCode=0
Nov 02 09:47:45.101573 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.094470    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r" event={"ID":"57cde931-be21-4545-bd3c-df85b7170196","Type":"ContainerDied","Data":"a76dfafb09a5ca42c060a2a5f115b02dbccf1d7b82b47876f7686c68d9c7b8d6"}
Nov 02 09:47:45.101573 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.094543    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r" event={"ID":"57cde931-be21-4545-bd3c-df85b7170196","Type":"ContainerDied","Data":"9ea58ba631f30734e673b7cf6438e79d91110acab593cc56f4431adf47afd9c6"}
Nov 02 09:47:45.101573 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.094572    4614 scope.go:122] "RemoveContainer" containerID="a76dfafb09a5ca42c060a2a5f115b02dbccf1d7b82b47876f7686c68d9c7b8d6"
Nov 02 09:47:45.101573 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.094422    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"
Nov 02 09:47:45.103621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.102500    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-model-cache\") pod \"57cde931-be21-4545-bd3c-df85b7170196\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") "
Nov 02 09:47:45.103621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.102577    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-dshm\") pod \"57cde931-be21-4545-bd3c-df85b7170196\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") "
Nov 02 09:47:45.103621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.102649    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57cde931-be21-4545-bd3c-df85b7170196-kube-api-access-lxjfd\" (UniqueName: \"kubernetes.io/projected/57cde931-be21-4545-bd3c-df85b7170196-kube-api-access-lxjfd\") pod \"57cde931-be21-4545-bd3c-df85b7170196\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") "
Nov 02 09:47:45.103621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.102706    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-tmp-dir\") pod \"57cde931-be21-4545-bd3c-df85b7170196\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") "
Nov 02 09:47:45.103621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.102777    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-home\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-home\") pod \"57cde931-be21-4545-bd3c-df85b7170196\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") "
Nov 02 09:47:45.103621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.102839    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57cde931-be21-4545-bd3c-df85b7170196-tls-certs\" (UniqueName: \"kubernetes.io/secret/57cde931-be21-4545-bd3c-df85b7170196-tls-certs\") pod \"57cde931-be21-4545-bd3c-df85b7170196\" (UID: \"57cde931-be21-4545-bd3c-df85b7170196\") "
Nov 02 09:47:45.104628 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.104575    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-home" pod "57cde931-be21-4545-bd3c-df85b7170196" (UID: "57cde931-be21-4545-bd3c-df85b7170196"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:47:45.105427 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.105330    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-model-cache" pod "57cde931-be21-4545-bd3c-df85b7170196" (UID: "57cde931-be21-4545-bd3c-df85b7170196"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:47:45.105689 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.105668    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-tmp-dir" pod "57cde931-be21-4545-bd3c-df85b7170196" (UID: "57cde931-be21-4545-bd3c-df85b7170196"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:47:45.147558 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.147520    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57cde931-be21-4545-bd3c-df85b7170196-tls-certs" pod "57cde931-be21-4545-bd3c-df85b7170196" (UID: "57cde931-be21-4545-bd3c-df85b7170196"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:47:45.161736 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.161712    4614 scope.go:122] "RemoveContainer" containerID="a76dfafb09a5ca42c060a2a5f115b02dbccf1d7b82b47876f7686c68d9c7b8d6"
Nov 02 09:47:45.165097 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:47:45.165068    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76dfafb09a5ca42c060a2a5f115b02dbccf1d7b82b47876f7686c68d9c7b8d6\": container with ID starting with a76dfafb09a5ca42c060a2a5f115b02dbccf1d7b82b47876f7686c68d9c7b8d6 not found: ID does not exist" containerID="a76dfafb09a5ca42c060a2a5f115b02dbccf1d7b82b47876f7686c68d9c7b8d6"
Nov 02 09:47:45.165278 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.165248    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76dfafb09a5ca42c060a2a5f115b02dbccf1d7b82b47876f7686c68d9c7b8d6"} err="failed to get container status \"a76dfafb09a5ca42c060a2a5f115b02dbccf1d7b82b47876f7686c68d9c7b8d6\": rpc error: code = NotFound desc = could not find container \"a76dfafb09a5ca42c060a2a5f115b02dbccf1d7b82b47876f7686c68d9c7b8d6\": container with ID starting with a76dfafb09a5ca42c060a2a5f115b02dbccf1d7b82b47876f7686c68d9c7b8d6 not found: ID does not exist"
Nov 02 09:47:45.167907 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.167862    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:45.168136 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.168113    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:45.168330 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.168295    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:45.205494 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.205394    4614 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-tmp-dir\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:45.205494 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.205422    4614 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-home\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:45.205494 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.205437    4614 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cde931-be21-4545-bd3c-df85b7170196-tls-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:45.205494 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.205450    4614 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-model-cache\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:45.239977 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.239918    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cde931-be21-4545-bd3c-df85b7170196-kube-api-access-lxjfd" pod "57cde931-be21-4545-bd3c-df85b7170196" (UID: "57cde931-be21-4545-bd3c-df85b7170196"). InnerVolumeSpecName "kube-api-access-lxjfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:45.247858 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.247804    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-dshm" pod "57cde931-be21-4545-bd3c-df85b7170196" (UID: "57cde931-be21-4545-bd3c-df85b7170196"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:47:45.308276 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.308239    4614 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cde931-be21-4545-bd3c-df85b7170196-dshm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:45.312483 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.312460    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxjfd\" (UniqueName: \"kubernetes.io/projected/57cde931-be21-4545-bd3c-df85b7170196-kube-api-access-lxjfd\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:45.558664 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.558592    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r" podUID="57cde931-be21-4545-bd3c-df85b7170196" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:45.603435 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.603396    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"]
Nov 02 09:47:45.619804 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.619767    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-bde944/auth-test-model-bde944-kserve-5945994c4f-rhl2r"]
Nov 02 09:47:45.994595 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.994561    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 02 09:47:46.003975 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:45.994882    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" podUID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 02 09:47:46.032391 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:46.022919    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 09:47:46.032391 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:46.022977    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 09:47:46.258007 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:46.257917    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd" event={"ID":"a5ed6879-0ce6-4705-a1bc-eefbfef9157f","Type":"ContainerStarted","Data":"9a4878af8a305f63a68fbede3104f2dd2da02360caf6b238e2c4343d2c694363"}
Nov 02 09:47:46.258599 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:46.258571    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd" event={"ID":"a5ed6879-0ce6-4705-a1bc-eefbfef9157f","Type":"ContainerStarted","Data":"06e6b969382a62622b03ad7e1e7b38b64bde75a0d703b507d9c50daaa0b823eb"}
Nov 02 09:47:47.020378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:47.019436    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:47.132295 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:47.132245    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:47.197542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:47.197491    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:47:47.241155 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:47.241121    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:47.302602 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:47.302418    4614 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 09:47:47.328556 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:47.328510    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd" podUID="a5ed6879-0ce6-4705-a1bc-eefbfef9157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 09:47:47.346505 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:47.346442    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492" podUID="ae2706aa-d41a-463a-acf6-79c6b2705f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:47.433476 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:47.433442    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57cde931-be21-4545-bd3c-df85b7170196" path="/var/lib/kubelet/pods/57cde931-be21-4545-bd3c-df85b7170196/volumes"
Nov 02 09:47:47.846521 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:47.846483    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:47:47.846812 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:47.846781    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:47.881229 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:47.881196    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:47:47.881539 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:47.881512    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" podUID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:48.032378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:48.022452    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:48.032378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:48.023248    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:47:48.035464 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:48.035430    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:47:48.035675 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:48.035648    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:48.048425 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:48.046831    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:47:48.048425 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:48.046888    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" podUID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:48.061380 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:48.059006    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd" podStartSLOduration=5.058970608 podStartE2EDuration="5.058970608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:46.306022723 +0000 UTC m=+747.138680807" watchObservedRunningTime="2026-11-02 09:47:48.058970608 +0000 UTC m=+748.891628689"
Nov 02 09:47:48.065376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:48.061938    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:48.237443 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:48.232465    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:48.238233 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:48.238204    4614 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 09:47:48.260742 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:48.260697    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd" podUID="a5ed6879-0ce6-4705-a1bc-eefbfef9157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 09:47:48.623752 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:48.623659    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:47:49.056377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:49.055495    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:47:49.056377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:49.056122    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" podUID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:49.081844 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:49.081789    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:47:49.082040 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:49.081867    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:49.219426 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:49.218515    4614 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 09:47:49.219426 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:49.218595    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd" podUID="a5ed6879-0ce6-4705-a1bc-eefbfef9157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 09:47:49.242838 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:49.242798    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:49.244802 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:49.244782    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:49.250031 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:49.249972    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:47:49.298621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:49.297139    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:49.299750 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:49.299725    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:49.316790 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:49.316661    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:47:50.214385 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:50.214329    4614 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 09:47:50.214769 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:50.214395    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd" podUID="a5ed6879-0ce6-4705-a1bc-eefbfef9157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 09:47:50.957927 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:50.957896    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"
Nov 02 09:47:50.963722 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:50.963680    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl" podUID="623fd585-5bd4-4a62-87fb-6c27d59bb0c0" containerName="main" containerID="cri-o://a4fc390a694c6f0414fdf77ea82d29f4dd896eacfe50d836bd3c34ccd2e45727" gracePeriod=45
Nov 02 09:47:51.221424 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.216460    4614 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 09:47:51.221424 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.216680    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd" podUID="a5ed6879-0ce6-4705-a1bc-eefbfef9157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 09:47:51.367294 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.367267    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:47:51.459499 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.459459    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-kube-api-access-m4j58\" (UniqueName: \"kubernetes.io/projected/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-kube-api-access-m4j58\") pod \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") "
Nov 02 09:47:51.459676 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.459504    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-home\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-home\") pod \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") "
Nov 02 09:47:51.459676 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.459555    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-tls-certs\") pod \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") "
Nov 02 09:47:51.459676 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.459581    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-tmp-dir\") pod \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") "
Nov 02 09:47:51.459676 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.459627    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-dshm\") pod \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") "
Nov 02 09:47:51.459897 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.459684    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-model-cache\") pod \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\" (UID: \"623fd585-5bd4-4a62-87fb-6c27d59bb0c0\") "
Nov 02 09:47:51.459897 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.459838    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-home" pod "623fd585-5bd4-4a62-87fb-6c27d59bb0c0" (UID: "623fd585-5bd4-4a62-87fb-6c27d59bb0c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:47:51.459897 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.459881    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-tmp-dir" pod "623fd585-5bd4-4a62-87fb-6c27d59bb0c0" (UID: "623fd585-5bd4-4a62-87fb-6c27d59bb0c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:47:51.460046 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.460018    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-model-cache" pod "623fd585-5bd4-4a62-87fb-6c27d59bb0c0" (UID: "623fd585-5bd4-4a62-87fb-6c27d59bb0c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:47:51.460232 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.460212    4614 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-home\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:51.460302 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.460235    4614 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-tmp-dir\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:51.460302 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.460249    4614 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-model-cache\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:51.462516 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.462114    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-tls-certs" pod "623fd585-5bd4-4a62-87fb-6c27d59bb0c0" (UID: "623fd585-5bd4-4a62-87fb-6c27d59bb0c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:47:51.462516 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.462167    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-dshm" pod "623fd585-5bd4-4a62-87fb-6c27d59bb0c0" (UID: "623fd585-5bd4-4a62-87fb-6c27d59bb0c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:47:51.462516 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.462179    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-kube-api-access-m4j58" pod "623fd585-5bd4-4a62-87fb-6c27d59bb0c0" (UID: "623fd585-5bd4-4a62-87fb-6c27d59bb0c0"). InnerVolumeSpecName "kube-api-access-m4j58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:51.470079 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.470052    4614 generic.go:356] "Generic (PLEG): container finished" podID="623fd585-5bd4-4a62-87fb-6c27d59bb0c0" containerID="a4fc390a694c6f0414fdf77ea82d29f4dd896eacfe50d836bd3c34ccd2e45727" exitCode=0
Nov 02 09:47:51.470199 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.470101    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"
Nov 02 09:47:51.470199 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.470137    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl" event={"ID":"623fd585-5bd4-4a62-87fb-6c27d59bb0c0","Type":"ContainerDied","Data":"a4fc390a694c6f0414fdf77ea82d29f4dd896eacfe50d836bd3c34ccd2e45727"}
Nov 02 09:47:51.470199 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.470183    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl" event={"ID":"623fd585-5bd4-4a62-87fb-6c27d59bb0c0","Type":"ContainerDied","Data":"76b5598e0e8746d7e7557c485a859d5224de2863291cc85c724a81726ebffe78"}
Nov 02 09:47:51.470305 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.470200    4614 scope.go:122] "RemoveContainer" containerID="a4fc390a694c6f0414fdf77ea82d29f4dd896eacfe50d836bd3c34ccd2e45727"
Nov 02 09:47:51.489517 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.489500    4614 scope.go:122] "RemoveContainer" containerID="a4fc390a694c6f0414fdf77ea82d29f4dd896eacfe50d836bd3c34ccd2e45727"
Nov 02 09:47:51.489815 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:47:51.489794    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fc390a694c6f0414fdf77ea82d29f4dd896eacfe50d836bd3c34ccd2e45727\": container with ID starting with a4fc390a694c6f0414fdf77ea82d29f4dd896eacfe50d836bd3c34ccd2e45727 not found: ID does not exist" containerID="a4fc390a694c6f0414fdf77ea82d29f4dd896eacfe50d836bd3c34ccd2e45727"
Nov 02 09:47:51.489889 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.489821    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fc390a694c6f0414fdf77ea82d29f4dd896eacfe50d836bd3c34ccd2e45727"} err="failed to get container status \"a4fc390a694c6f0414fdf77ea82d29f4dd896eacfe50d836bd3c34ccd2e45727\": rpc error: code = NotFound desc = could not find container \"a4fc390a694c6f0414fdf77ea82d29f4dd896eacfe50d836bd3c34ccd2e45727\": container with ID starting with a4fc390a694c6f0414fdf77ea82d29f4dd896eacfe50d836bd3c34ccd2e45727 not found: ID does not exist"
Nov 02 09:47:51.501063 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.501039    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"]
Nov 02 09:47:51.504885 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.504864    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9e7156/auth-test-model-9e7156-kserve-7f77bf9477-sl6fl"]
Nov 02 09:47:51.561235 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.561200    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4j58\" (UniqueName: \"kubernetes.io/projected/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-kube-api-access-m4j58\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:51.561235 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.561231    4614 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-tls-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:51.561476 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.561244    4614 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/623fd585-5bd4-4a62-87fb-6c27d59bb0c0-dshm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:51.955717 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:51.955648    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"
Nov 02 09:47:52.214644 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.214555    4614 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 09:47:52.214644 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.214621    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd" podUID="a5ed6879-0ce6-4705-a1bc-eefbfef9157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 09:47:52.405395 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.405343    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"]
Nov 02 09:47:52.405928 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.405913    4614 cpu_manager.go:414] "containerMap: removing container" podUID="623fd585-5bd4-4a62-87fb-6c27d59bb0c0" containerName="main"
Nov 02 09:47:52.405978 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.405930    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="623fd585-5bd4-4a62-87fb-6c27d59bb0c0" containerName="main"
Nov 02 09:47:52.405978 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.405954    4614 cpu_manager.go:414] "containerMap: removing container" podUID="57cde931-be21-4545-bd3c-df85b7170196" containerName="main"
Nov 02 09:47:52.405978 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.405960    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57cde931-be21-4545-bd3c-df85b7170196" containerName="main"
Nov 02 09:47:52.406076 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.406046    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="623fd585-5bd4-4a62-87fb-6c27d59bb0c0" containerName="main"
Nov 02 09:47:52.406076 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.406058    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="57cde931-be21-4545-bd3c-df85b7170196" containerName="main"
Nov 02 09:47:52.410669 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.410652    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"
Nov 02 09:47:52.415441 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.415420    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:47:52.415599 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.415432    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:47:52.418856 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.418833    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"]
Nov 02 09:47:52.477981 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.477946    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c18ead-e49f-46a8-8dc8-7a652bd8c774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-164501f8-sggxm\" (UID: \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"
Nov 02 09:47:52.478177 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.478033    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c18ead-e49f-46a8-8dc8-7a652bd8c774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-164501f8-sggxm\" (UID: \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"
Nov 02 09:47:52.478177 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.478131    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqf24\" (UniqueName: \"kubernetes.io/projected/92c18ead-e49f-46a8-8dc8-7a652bd8c774-kube-api-access-nqf24\") pod \"maas-api-revoke-sub-164501f8-sggxm\" (UID: \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"
Nov 02 09:47:52.579587 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.579549    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c18ead-e49f-46a8-8dc8-7a652bd8c774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-164501f8-sggxm\" (UID: \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"
Nov 02 09:47:52.579587 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.579591    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c18ead-e49f-46a8-8dc8-7a652bd8c774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-164501f8-sggxm\" (UID: \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"
Nov 02 09:47:52.579805 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.579662    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqf24\" (UniqueName: \"kubernetes.io/projected/92c18ead-e49f-46a8-8dc8-7a652bd8c774-kube-api-access-nqf24\") pod \"maas-api-revoke-sub-164501f8-sggxm\" (UID: \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"
Nov 02 09:47:52.580423 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.580399    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c18ead-e49f-46a8-8dc8-7a652bd8c774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-164501f8-sggxm\" (UID: \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"
Nov 02 09:47:52.582136 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.582092    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c18ead-e49f-46a8-8dc8-7a652bd8c774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-164501f8-sggxm\" (UID: \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"
Nov 02 09:47:52.591905 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.591884    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqf24\" (UniqueName: \"kubernetes.io/projected/92c18ead-e49f-46a8-8dc8-7a652bd8c774-kube-api-access-nqf24\") pod \"maas-api-revoke-sub-164501f8-sggxm\" (UID: \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"
Nov 02 09:47:52.690222 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.690187    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"]
Nov 02 09:47:52.695330 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.695296    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.705328 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.705133    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"]
Nov 02 09:47:52.752424 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.747558    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"
Nov 02 09:47:52.781435 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.781404    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.781588 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.781461    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.781588 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.781545    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.781588 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.781570    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.781773 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.781622    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.781773 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.781655    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vpw9\" (UniqueName: \"kubernetes.io/projected/b807678e-0dc1-4d67-9ead-83414e09f53d-kube-api-access-9vpw9\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.781773 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.781695    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.781930 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.781777    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.781930 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.781808    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b807678e-0dc1-4d67-9ead-83414e09f53d-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.903189 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.902470    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.903189 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.902531    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.903189 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.902642    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.903189 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.902673    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.903189 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.902730    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.903189 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.902766    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vpw9\" (UniqueName: \"kubernetes.io/projected/b807678e-0dc1-4d67-9ead-83414e09f53d-kube-api-access-9vpw9\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.903189 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.902810    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.903189 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.902912    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.903189 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.902948    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b807678e-0dc1-4d67-9ead-83414e09f53d-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.903940 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.903892    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b807678e-0dc1-4d67-9ead-83414e09f53d-istiod-ca-cert\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.904434 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.904413    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-workload-certs\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.904852 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.904833    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-data\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.905498 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.905292    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-workload-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.919401 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.913732    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-credential-socket\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.925326 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.925297    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-envoy\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.953375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.952920    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-podinfo\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.955513 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.955325    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-token\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:52.957892 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:52.957866    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vpw9\" (UniqueName: \"kubernetes.io/projected/b807678e-0dc1-4d67-9ead-83414e09f53d-kube-api-access-9vpw9\") pod \"e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") " pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:53.013390 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.013305    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:53.061213 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.061172    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"]
Nov 02 09:47:53.073552 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:47:53.073527    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c18ead_e49f_46a8_8dc8_7a652bd8c774.slice/crio-a19042b49a1770b910dcb9737553c39e993a7e2eecd837e1214020e5e1361ee7 WatchSource:0}: Error finding container a19042b49a1770b910dcb9737553c39e993a7e2eecd837e1214020e5e1361ee7: Status 404 returned error can't find the container with id a19042b49a1770b910dcb9737553c39e993a7e2eecd837e1214020e5e1361ee7
Nov 02 09:47:53.214777 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.214745    4614 patch_prober.go:29] interesting pod/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 09:47:53.214952 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.214798    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd" podUID="a5ed6879-0ce6-4705-a1bc-eefbfef9157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 09:47:53.257238 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.257197    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"]
Nov 02 09:47:53.265769 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.265699    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"]
Nov 02 09:47:53.265917 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.265809    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.269579 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.269551    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8743a5-openshift-default-dockercfg-6pdx6\""
Nov 02 09:47:53.309054 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.309022    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-podinfo\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.309205 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.309177    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8sqd\" (UniqueName: \"kubernetes.io/projected/270f7540-a28f-446a-bc7f-3b3d143bc351-kube-api-access-k8sqd\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.309296 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.309273    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-workload-certs\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.309335 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.309319    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-workload-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.309420 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.309408    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-token\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.309468 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.309431    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-data\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.309532 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.309518    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270f7540-a28f-446a-bc7f-3b3d143bc351-istiod-ca-cert\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.309570 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.309541    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-envoy\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.309624 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.309578    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-credential-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.316030 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.316008    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="623fd585-5bd4-4a62-87fb-6c27d59bb0c0" path="/var/lib/kubelet/pods/623fd585-5bd4-4a62-87fb-6c27d59bb0c0/volumes"
Nov 02 09:47:53.350128 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.349704    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"]
Nov 02 09:47:53.352753 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:47:53.352727    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb807678e_0dc1_4d67_9ead_83414e09f53d.slice/crio-16c4cd2403881c32669b49f906c6b05b43f432e884c8e7e0474bdeed02f89b24 WatchSource:0}: Error finding container 16c4cd2403881c32669b49f906c6b05b43f432e884c8e7e0474bdeed02f89b24: Status 404 returned error can't find the container with id 16c4cd2403881c32669b49f906c6b05b43f432e884c8e7e0474bdeed02f89b24
Nov 02 09:47:53.355391 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.355343    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:53.355487 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.355426    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:53.355487 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.355455    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:53.410971 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.410944    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8sqd\" (UniqueName: \"kubernetes.io/projected/270f7540-a28f-446a-bc7f-3b3d143bc351-kube-api-access-k8sqd\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.411279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.410993    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-workload-certs\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.411279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.411024    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-workload-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.411279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.411054    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-token\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.411279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.411081    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-data\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.411279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.411162    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270f7540-a28f-446a-bc7f-3b3d143bc351-istiod-ca-cert\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.411279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.411189    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-envoy\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.411279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.411250    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-credential-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.411697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.411287    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-podinfo\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.411697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.411527    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-workload-certs\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.411697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.411632    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-data\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.411905 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.411850    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-credential-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.411905 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.411897    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-workload-socket\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.412150 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.412122    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270f7540-a28f-446a-bc7f-3b3d143bc351-istiod-ca-cert\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.413529 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.413508    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-envoy\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.413981 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.413955    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-podinfo\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.423438 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.423413    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-token\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.423568 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.423445    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8sqd\" (UniqueName: \"kubernetes.io/projected/270f7540-a28f-446a-bc7f-3b3d143bc351-kube-api-access-k8sqd\") pod \"e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") " pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.489161 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.488884    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm" event={"ID":"92c18ead-e49f-46a8-8dc8-7a652bd8c774","Type":"ContainerStarted","Data":"54ecee23272c2c550add6fc3abac5562220c83c3212c7c9d1b6cf79c88bd3997"}
Nov 02 09:47:53.489161 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.488939    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm" event={"ID":"92c18ead-e49f-46a8-8dc8-7a652bd8c774","Type":"ContainerStarted","Data":"a19042b49a1770b910dcb9737553c39e993a7e2eecd837e1214020e5e1361ee7"}
Nov 02 09:47:53.490987 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.490950    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" event={"ID":"b807678e-0dc1-4d67-9ead-83414e09f53d","Type":"ContainerStarted","Data":"f001a6c0f4336d5bfcd40ef5f82c91eb7712da674aa4e8f4ba70dd5822227d68"}
Nov 02 09:47:53.491105 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.491011    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" event={"ID":"b807678e-0dc1-4d67-9ead-83414e09f53d","Type":"ContainerStarted","Data":"16c4cd2403881c32669b49f906c6b05b43f432e884c8e7e0474bdeed02f89b24"}
Nov 02 09:47:53.509405 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.509322    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm" podStartSLOduration=1.509302589 podStartE2EDuration="1.509302589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:53.506743508 +0000 UTC m=+754.339401589" watchObservedRunningTime="2026-11-02 09:47:53.509302589 +0000 UTC m=+754.341960731"
Nov 02 09:47:53.534606 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.534494    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" podStartSLOduration=1.5344738530000002 podStartE2EDuration="1.534473853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:53.528476211 +0000 UTC m=+754.361134290" watchObservedRunningTime="2026-11-02 09:47:53.534473853 +0000 UTC m=+754.367131934"
Nov 02 09:47:53.587416 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.587382    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:53.892605 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.892577    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"]
Nov 02 09:47:53.894122 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:47:53.894101    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270f7540_a28f_446a_bc7f_3b3d143bc351.slice/crio-9128ccd490161322e3b33ba7cb45ba4b85d2a10ba4f3f032446394c2adaf020c WatchSource:0}: Error finding container 9128ccd490161322e3b33ba7cb45ba4b85d2a10ba4f3f032446394c2adaf020c: Status 404 returned error can't find the container with id 9128ccd490161322e3b33ba7cb45ba4b85d2a10ba4f3f032446394c2adaf020c
Nov 02 09:47:53.907553 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.907501    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:53.907905 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.907883    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:53.908058 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:53.908039    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:47:54.017702 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:54.017662    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:47:54.052536 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:54.052478    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:47:54.052721 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:54.052588    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 09:47:54.488211 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:54.486175    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:54.488211 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:54.487618    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:54.492019 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:54.491993    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:47:54.539377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:54.533056    4614 generic.go:356] "Generic (PLEG): container finished" podID="92c18ead-e49f-46a8-8dc8-7a652bd8c774" containerID="54ecee23272c2c550add6fc3abac5562220c83c3212c7c9d1b6cf79c88bd3997" exitCode=0
Nov 02 09:47:54.539377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:54.533139    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm" event={"ID":"92c18ead-e49f-46a8-8dc8-7a652bd8c774","Type":"ContainerDied","Data":"54ecee23272c2c550add6fc3abac5562220c83c3212c7c9d1b6cf79c88bd3997"}
Nov 02 09:47:54.565493 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:54.553983    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm" event={"ID":"270f7540-a28f-446a-bc7f-3b3d143bc351","Type":"ContainerStarted","Data":"2fc1c464b1f5f9d7b6852b2d94d1a420d24d1b2c1aa66ea970cf93dcfc3f96ce"}
Nov 02 09:47:54.565493 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:54.554045    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm" event={"ID":"270f7540-a28f-446a-bc7f-3b3d143bc351","Type":"ContainerStarted","Data":"9128ccd490161322e3b33ba7cb45ba4b85d2a10ba4f3f032446394c2adaf020c"}
Nov 02 09:47:54.627716 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:54.595557    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:47:54.716202 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:54.715737    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm" podStartSLOduration=1.715719808 podStartE2EDuration="1.715719808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:47:54.683816446 +0000 UTC m=+755.516474528" watchObservedRunningTime="2026-11-02 09:47:54.715719808 +0000 UTC m=+755.548377888"
Nov 02 09:47:55.019606 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:55.019538    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:47:55.019899 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:55.019874    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 09:47:55.683382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:55.682503    4614 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 02 09:47:55.683382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:55.683080    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm" podUID="270f7540-a28f-446a-bc7f-3b3d143bc351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 02 09:47:55.900509 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:55.880094    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:56.026376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:56.023110    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:47:56.026376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:56.023183    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 09:47:56.232379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:56.231653    4614 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": context deadline exceeded"
Nov 02 09:47:56.274383 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:56.267667    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:56.609465 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:56.607768    4614 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 02 09:47:56.609465 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:56.607828    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm" podUID="270f7540-a28f-446a-bc7f-3b3d143bc351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 02 09:47:56.924462 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:56.910878    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:56.939383 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:56.929055    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:47:56.949215 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:56.949183    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"
Nov 02 09:47:57.131576 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.131475    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm" event={"ID":"92c18ead-e49f-46a8-8dc8-7a652bd8c774","Type":"ContainerDied","Data":"a19042b49a1770b910dcb9737553c39e993a7e2eecd837e1214020e5e1361ee7"}
Nov 02 09:47:57.131576 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.131541    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a19042b49a1770b910dcb9737553c39e993a7e2eecd837e1214020e5e1361ee7"
Nov 02 09:47:57.221430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.168545    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:47:57.221430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.168621    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 09:47:57.313379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.286461    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:57.313379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.287106    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:47:57.371393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.354831    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c18ead-e49f-46a8-8dc8-7a652bd8c774-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c18ead-e49f-46a8-8dc8-7a652bd8c774-maas-api-openshift-ca\") pod \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\" (UID: \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\") "
Nov 02 09:47:57.371393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.354981    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92c18ead-e49f-46a8-8dc8-7a652bd8c774-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c18ead-e49f-46a8-8dc8-7a652bd8c774-maas-api-xks-ca\") pod \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\" (UID: \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\") "
Nov 02 09:47:57.371393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.355039    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c18ead-e49f-46a8-8dc8-7a652bd8c774-kube-api-access-nqf24\" (UniqueName: \"kubernetes.io/projected/92c18ead-e49f-46a8-8dc8-7a652bd8c774-kube-api-access-nqf24\") pod \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\" (UID: \"92c18ead-e49f-46a8-8dc8-7a652bd8c774\") "
Nov 02 09:47:57.371393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.361562    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c18ead-e49f-46a8-8dc8-7a652bd8c774-maas-api-openshift-ca" pod "92c18ead-e49f-46a8-8dc8-7a652bd8c774" (UID: "92c18ead-e49f-46a8-8dc8-7a652bd8c774"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:47:57.502432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.485817    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c18ead-e49f-46a8-8dc8-7a652bd8c774-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:57.574748 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.570668    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c18ead-e49f-46a8-8dc8-7a652bd8c774-kube-api-access-nqf24" pod "92c18ead-e49f-46a8-8dc8-7a652bd8c774" (UID: "92c18ead-e49f-46a8-8dc8-7a652bd8c774"). InnerVolumeSpecName "kube-api-access-nqf24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:47:57.607873 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.607832    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92c18ead-e49f-46a8-8dc8-7a652bd8c774-maas-api-xks-ca" pod "92c18ead-e49f-46a8-8dc8-7a652bd8c774" (UID: "92c18ead-e49f-46a8-8dc8-7a652bd8c774"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:47:57.699377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.620256    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c18ead-e49f-46a8-8dc8-7a652bd8c774-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:57.699377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.620293    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqf24\" (UniqueName: \"kubernetes.io/projected/92c18ead-e49f-46a8-8dc8-7a652bd8c774-kube-api-access-nqf24\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:47:57.699377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.632844    4614 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 02 09:47:57.699377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.632924    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm" podUID="270f7540-a28f-446a-bc7f-3b3d143bc351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 02 09:47:57.764584 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:57.753767    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:47:58.021431 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:58.020484    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:47:58.021431 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:58.020573    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 09:47:58.141514 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:58.141463    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:58.272417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:58.272318    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"
Nov 02 09:47:58.415473 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:58.305747    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 02 09:47:58.428379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:58.305916    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:58.605152 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:58.605061    4614 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 02 09:47:58.618877 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:58.618826    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm" podUID="270f7540-a28f-446a-bc7f-3b3d143bc351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 02 09:47:58.750543 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:58.750488    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:59.021373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:59.013744    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:47:59.021373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:59.013808    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 09:47:59.572216 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:59.571458    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:59.593670 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:59.592474    4614 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 02 09:47:59.593670 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:59.592533    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm" podUID="270f7540-a28f-446a-bc7f-3b3d143bc351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 02 09:47:59.634195 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:59.632455    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 02 09:47:59.763376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:59.760308    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:47:59.763376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:47:59.761066    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:48:00.027646 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:00.025605    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:48:00.028256 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:00.028205    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 09:48:00.037152 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:00.037124    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:48:00.202374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:00.109682    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 02 09:48:00.436432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:00.420647    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:48:00.588495 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:00.588455    4614 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 02 09:48:00.588797 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:00.588765    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm" podUID="270f7540-a28f-446a-bc7f-3b3d143bc351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 02 09:48:01.014237 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:01.014197    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:48:01.014458 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:01.014278    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 09:48:01.598771 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:01.598734    4614 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 02 09:48:01.599316 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:01.599284    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm" podUID="270f7540-a28f-446a-bc7f-3b3d143bc351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 02 09:48:02.016868 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:02.016828    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:48:02.017473 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:02.017435    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 09:48:02.588779 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:02.588729    4614 patch_prober.go:29] interesting pod/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 02 09:48:02.589002 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:02.588792    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm" podUID="270f7540-a28f-446a-bc7f-3b3d143bc351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 02 09:48:03.014389 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:03.014337    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 02 09:48:03.014856 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:03.014811    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 02 09:48:04.119691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:04.118204    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:48:04.119691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:04.119564    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:48:04.120806 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:04.120760    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:48:05.062493 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:05.062454    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:48:05.062746 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:05.062716    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:48:05.793266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:05.793230    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:48:05.794643 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:05.794622    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:48:05.795907 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:05.795888    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:48:06.874374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:06.873485    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:48:06.874374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:06.873570    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm" podUID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:48:09.638749 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:48:09.638709    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:48:10.490447 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:10.490412    4614 generic.go:356] "Generic (PLEG): container finished" podID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerID="f9e7be1164127f4bdff0792e229b1baf8b546dc62c84e4ed84eb70b583652e26" exitCode=0
Nov 02 09:48:10.490697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:10.490470    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm" event={"ID":"a0a1b0ce-e12b-4227-90be-f17a736ab20d","Type":"ContainerDied","Data":"f9e7be1164127f4bdff0792e229b1baf8b546dc62c84e4ed84eb70b583652e26"}
Nov 02 09:48:10.490947 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:10.490929    4614 scope.go:122] "RemoveContainer" containerID="f9e7be1164127f4bdff0792e229b1baf8b546dc62c84e4ed84eb70b583652e26"
Nov 02 09:48:10.492587 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:10.492546    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:48:10.492703 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:10.492629    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:48:10.492703 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:10.492664    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:48:11.498485 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:11.498456    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm" event={"ID":"a0a1b0ce-e12b-4227-90be-f17a736ab20d","Type":"ContainerStarted","Data":"033e00d3038edac90628ea16925637a39ff4a0d03aa5cd2ece559f549b7b9e21"}
Nov 02 09:48:12.407208 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:12.407169    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:48:12.407487 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:12.407404    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 02 09:48:12.407487 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:12.407469    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm" podUID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 02 09:48:13.406706 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:13.406668    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 02 09:48:13.407096 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:13.406728    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm" podUID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 02 09:48:14.407027 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:14.406989    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 02 09:48:14.407435 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:14.407054    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm" podUID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 02 09:48:15.419371 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:15.407800    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 02 09:48:15.419371 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:15.407864    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm" podUID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 02 09:48:16.411376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:16.408755    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 02 09:48:16.411376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:16.408819    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm" podUID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 02 09:48:17.172385 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:17.145440    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:48:17.436444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:17.429438    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:48:17.679389 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:17.673463    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:48:18.063497 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:18.063452    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 02 09:48:18.324446 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:18.322801    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:48:18.324446 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:18.324107    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:48:18.436697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:18.436655    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 09:48:36.043377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:36.042444    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:48:36.489377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:36.488456    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:48:36.626542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:36.626477    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:48:36.832138 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:36.832004    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:48:36.832490 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:36.832450    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:48:37.044506 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:37.044449    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Nov 02 09:48:37.046289 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:37.046266    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:48:37.072460 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:37.072419    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:48:37.085425 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:37.085327    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:48:37.184486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:37.184443    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:48:39.637383 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:48:39.637334    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:48:48.139896 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.139863    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"]
Nov 02 09:48:48.140440 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.140421    4614 cpu_manager.go:414] "containerMap: removing container" podUID="92c18ead-e49f-46a8-8dc8-7a652bd8c774" containerName="revoke-keys"
Nov 02 09:48:48.140512 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.140443    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c18ead-e49f-46a8-8dc8-7a652bd8c774" containerName="revoke-keys"
Nov 02 09:48:48.140594 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.140583    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c18ead-e49f-46a8-8dc8-7a652bd8c774" containerName="revoke-keys"
Nov 02 09:48:48.144492 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.144477    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.157365 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.157329    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"routing-overlay\""
Nov 02 09:48:48.157501 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.157417    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-1521404c1537\""
Nov 02 09:48:48.157501 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.157329    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-4a99520c97db\""
Nov 02 09:48:48.157741 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.157718    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"kube-root-ca.crt\""
Nov 02 09:48:48.158720 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.158699    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-plugins-e2e-extmod-b-204fba\""
Nov 02 09:48:48.158876 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.158771    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"openshift-service-ca.crt\""
Nov 02 09:48:48.158876 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.158863    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-e2e--e7669bbb-dockercfg-zngpv\""
Nov 02 09:48:48.162013 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.161985    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"]
Nov 02 09:48:48.190867 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.190835    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86364d67-8d18-4d74-94ee-b03b7bbe399f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.191038 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.190882    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86364d67-8d18-4d74-94ee-b03b7bbe399f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.191038 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.190953    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8gj\" (UniqueName: \"kubernetes.io/projected/86364d67-8d18-4d74-94ee-b03b7bbe399f-kube-api-access-jg8gj\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.191161 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.191045    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86364d67-8d18-4d74-94ee-b03b7bbe399f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.291766 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.291731    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86364d67-8d18-4d74-94ee-b03b7bbe399f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.291954 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.291783    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86364d67-8d18-4d74-94ee-b03b7bbe399f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.291954 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.291807    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8gj\" (UniqueName: \"kubernetes.io/projected/86364d67-8d18-4d74-94ee-b03b7bbe399f-kube-api-access-jg8gj\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.291954 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.291843    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86364d67-8d18-4d74-94ee-b03b7bbe399f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.292505 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.292480    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86364d67-8d18-4d74-94ee-b03b7bbe399f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.292609 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.292591    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86364d67-8d18-4d74-94ee-b03b7bbe399f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.294303 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.294283    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86364d67-8d18-4d74-94ee-b03b7bbe399f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.311879 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.311849    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8gj\" (UniqueName: \"kubernetes.io/projected/86364d67-8d18-4d74-94ee-b03b7bbe399f-kube-api-access-jg8gj\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.462179 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.462024    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.743773 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.743744    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"]
Nov 02 09:48:48.744588 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:48:48.744554    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86364d67_8d18_4d74_94ee_b03b7bbe399f.slice/crio-d0053486f1cf457efed2b9947d992d35a4edd694eceeb27632d6fa6d1f5fb8a8 WatchSource:0}: Error finding container d0053486f1cf457efed2b9947d992d35a4edd694eceeb27632d6fa6d1f5fb8a8: Status 404 returned error can't find the container with id d0053486f1cf457efed2b9947d992d35a4edd694eceeb27632d6fa6d1f5fb8a8
Nov 02 09:48:48.964003 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.963965    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv" event={"ID":"86364d67-8d18-4d74-94ee-b03b7bbe399f","Type":"ContainerStarted","Data":"9849fe3addb3deeeeba7793e8c43267ce378f0c5ef7db067ee05a6543bc74a61"}
Nov 02 09:48:48.964218 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.964026    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv" event={"ID":"86364d67-8d18-4d74-94ee-b03b7bbe399f","Type":"ContainerStarted","Data":"d0053486f1cf457efed2b9947d992d35a4edd694eceeb27632d6fa6d1f5fb8a8"}
Nov 02 09:48:48.964218 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.964169    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:48:48.984294 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:48.984244    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv" podStartSLOduration=0.984229812 podStartE2EDuration="984.229812ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:48:48.984004613 +0000 UTC m=+809.816662695" watchObservedRunningTime="2026-11-02 09:48:48.984229812 +0000 UTC m=+809.816887893"
Nov 02 09:48:59.979224 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:48:59.979192    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:49:01.391496 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.391449    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"]
Nov 02 09:49:01.396448 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.396428    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"
Nov 02 09:49:01.403170 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.403146    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:49:01.403327 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.403224    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:49:01.406437 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.406412    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"]
Nov 02 09:49:01.421222 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.421197    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afacc90-28a8-43ff-9629-992ee73f304b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be24fe9a-f8g4c\" (UID: \"7afacc90-28a8-43ff-9629-992ee73f304b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"
Nov 02 09:49:01.421375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.421327    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwrd6\" (UniqueName: \"kubernetes.io/projected/7afacc90-28a8-43ff-9629-992ee73f304b-kube-api-access-bwrd6\") pod \"maas-api-revoke-sub-be24fe9a-f8g4c\" (UID: \"7afacc90-28a8-43ff-9629-992ee73f304b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"
Nov 02 09:49:01.421512 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.421489    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afacc90-28a8-43ff-9629-992ee73f304b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be24fe9a-f8g4c\" (UID: \"7afacc90-28a8-43ff-9629-992ee73f304b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"
Nov 02 09:49:01.522384 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.522322    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afacc90-28a8-43ff-9629-992ee73f304b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be24fe9a-f8g4c\" (UID: \"7afacc90-28a8-43ff-9629-992ee73f304b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"
Nov 02 09:49:01.522562 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.522436    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afacc90-28a8-43ff-9629-992ee73f304b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be24fe9a-f8g4c\" (UID: \"7afacc90-28a8-43ff-9629-992ee73f304b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"
Nov 02 09:49:01.522562 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.522492    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrd6\" (UniqueName: \"kubernetes.io/projected/7afacc90-28a8-43ff-9629-992ee73f304b-kube-api-access-bwrd6\") pod \"maas-api-revoke-sub-be24fe9a-f8g4c\" (UID: \"7afacc90-28a8-43ff-9629-992ee73f304b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"
Nov 02 09:49:01.523141 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.523116    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afacc90-28a8-43ff-9629-992ee73f304b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be24fe9a-f8g4c\" (UID: \"7afacc90-28a8-43ff-9629-992ee73f304b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"
Nov 02 09:49:01.524969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.524945    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afacc90-28a8-43ff-9629-992ee73f304b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be24fe9a-f8g4c\" (UID: \"7afacc90-28a8-43ff-9629-992ee73f304b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"
Nov 02 09:49:01.536420 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.536395    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwrd6\" (UniqueName: \"kubernetes.io/projected/7afacc90-28a8-43ff-9629-992ee73f304b-kube-api-access-bwrd6\") pod \"maas-api-revoke-sub-be24fe9a-f8g4c\" (UID: \"7afacc90-28a8-43ff-9629-992ee73f304b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"
Nov 02 09:49:01.712974 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:01.712581    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"
Nov 02 09:49:02.044161 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:02.044129    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"]
Nov 02 09:49:02.045634 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:49:02.045598    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afacc90_28a8_43ff_9629_992ee73f304b.slice/crio-a357551e204f893b923a10e2b48e92225d5dcd5ee71fff050bc033513c1e8908 WatchSource:0}: Error finding container a357551e204f893b923a10e2b48e92225d5dcd5ee71fff050bc033513c1e8908: Status 404 returned error can't find the container with id a357551e204f893b923a10e2b48e92225d5dcd5ee71fff050bc033513c1e8908
Nov 02 09:49:02.101253 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:02.101228    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c" event={"ID":"7afacc90-28a8-43ff-9629-992ee73f304b","Type":"ContainerStarted","Data":"a357551e204f893b923a10e2b48e92225d5dcd5ee71fff050bc033513c1e8908"}
Nov 02 09:49:03.109272 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:03.109237    4614 generic.go:356] "Generic (PLEG): container finished" podID="7afacc90-28a8-43ff-9629-992ee73f304b" containerID="3f41c935363e047a5fea6f875472534a6f6305abd6e0d676ff622ee4d0c07a56" exitCode=0
Nov 02 09:49:03.109743 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:03.109311    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c" event={"ID":"7afacc90-28a8-43ff-9629-992ee73f304b","Type":"ContainerDied","Data":"3f41c935363e047a5fea6f875472534a6f6305abd6e0d676ff622ee4d0c07a56"}
Nov 02 09:49:04.301982 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:04.301958    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"
Nov 02 09:49:04.381227 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:04.381148    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7afacc90-28a8-43ff-9629-992ee73f304b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afacc90-28a8-43ff-9629-992ee73f304b-maas-api-xks-ca\") pod \"7afacc90-28a8-43ff-9629-992ee73f304b\" (UID: \"7afacc90-28a8-43ff-9629-992ee73f304b\") "
Nov 02 09:49:04.381427 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:04.381290    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7afacc90-28a8-43ff-9629-992ee73f304b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afacc90-28a8-43ff-9629-992ee73f304b-maas-api-openshift-ca\") pod \"7afacc90-28a8-43ff-9629-992ee73f304b\" (UID: \"7afacc90-28a8-43ff-9629-992ee73f304b\") "
Nov 02 09:49:04.381427 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:04.381327    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7afacc90-28a8-43ff-9629-992ee73f304b-kube-api-access-bwrd6\" (UniqueName: \"kubernetes.io/projected/7afacc90-28a8-43ff-9629-992ee73f304b-kube-api-access-bwrd6\") pod \"7afacc90-28a8-43ff-9629-992ee73f304b\" (UID: \"7afacc90-28a8-43ff-9629-992ee73f304b\") "
Nov 02 09:49:04.381742 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:04.381709    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7afacc90-28a8-43ff-9629-992ee73f304b-maas-api-openshift-ca" pod "7afacc90-28a8-43ff-9629-992ee73f304b" (UID: "7afacc90-28a8-43ff-9629-992ee73f304b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:04.381888 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:04.381873    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afacc90-28a8-43ff-9629-992ee73f304b-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:04.383290 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:04.383267    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7afacc90-28a8-43ff-9629-992ee73f304b-maas-api-xks-ca" pod "7afacc90-28a8-43ff-9629-992ee73f304b" (UID: "7afacc90-28a8-43ff-9629-992ee73f304b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:49:04.383537 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:04.383482    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afacc90-28a8-43ff-9629-992ee73f304b-kube-api-access-bwrd6" pod "7afacc90-28a8-43ff-9629-992ee73f304b" (UID: "7afacc90-28a8-43ff-9629-992ee73f304b"). InnerVolumeSpecName "kube-api-access-bwrd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:04.483184 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:04.483157    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afacc90-28a8-43ff-9629-992ee73f304b-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:04.483184 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:04.483181    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwrd6\" (UniqueName: \"kubernetes.io/projected/7afacc90-28a8-43ff-9629-992ee73f304b-kube-api-access-bwrd6\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:05.123101 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:05.123070    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"
Nov 02 09:49:05.123101 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:05.123095    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c" event={"ID":"7afacc90-28a8-43ff-9629-992ee73f304b","Type":"ContainerDied","Data":"a357551e204f893b923a10e2b48e92225d5dcd5ee71fff050bc033513c1e8908"}
Nov 02 09:49:05.123319 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:05.123145    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a357551e204f893b923a10e2b48e92225d5dcd5ee71fff050bc033513c1e8908"
Nov 02 09:49:09.711391 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:09.711329    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:49:14.573966 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.573927    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"]
Nov 02 09:49:14.574635 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.574615    4614 cpu_manager.go:414] "containerMap: removing container" podUID="7afacc90-28a8-43ff-9629-992ee73f304b" containerName="revoke-keys"
Nov 02 09:49:14.574745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.574636    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7afacc90-28a8-43ff-9629-992ee73f304b" containerName="revoke-keys"
Nov 02 09:49:14.574839 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.574826    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="7afacc90-28a8-43ff-9629-992ee73f304b" containerName="revoke-keys"
Nov 02 09:49:14.578445 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.578423    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"
Nov 02 09:49:14.583437 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.583414    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:49:14.584462 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.584440    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:49:14.587894 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.587873    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"]
Nov 02 09:49:14.668841 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.668807    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994b0fe2-4573-4250-929e-0cd7d12e2070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e8765b-gb75g\" (UID: \"994b0fe2-4573-4250-929e-0cd7d12e2070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"
Nov 02 09:49:14.669015 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.668933    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994b0fe2-4573-4250-929e-0cd7d12e2070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e8765b-gb75g\" (UID: \"994b0fe2-4573-4250-929e-0cd7d12e2070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"
Nov 02 09:49:14.669015 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.668991    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5xwj\" (UniqueName: \"kubernetes.io/projected/994b0fe2-4573-4250-929e-0cd7d12e2070-kube-api-access-r5xwj\") pod \"maas-api-revoke-sub-c2e8765b-gb75g\" (UID: \"994b0fe2-4573-4250-929e-0cd7d12e2070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"
Nov 02 09:49:14.774300 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.774262    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994b0fe2-4573-4250-929e-0cd7d12e2070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e8765b-gb75g\" (UID: \"994b0fe2-4573-4250-929e-0cd7d12e2070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"
Nov 02 09:49:14.780793 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.777857    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xwj\" (UniqueName: \"kubernetes.io/projected/994b0fe2-4573-4250-929e-0cd7d12e2070-kube-api-access-r5xwj\") pod \"maas-api-revoke-sub-c2e8765b-gb75g\" (UID: \"994b0fe2-4573-4250-929e-0cd7d12e2070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"
Nov 02 09:49:14.783230 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.783078    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994b0fe2-4573-4250-929e-0cd7d12e2070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2e8765b-gb75g\" (UID: \"994b0fe2-4573-4250-929e-0cd7d12e2070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"
Nov 02 09:49:14.783475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.783307    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994b0fe2-4573-4250-929e-0cd7d12e2070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e8765b-gb75g\" (UID: \"994b0fe2-4573-4250-929e-0cd7d12e2070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"
Nov 02 09:49:14.792845 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.792819    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994b0fe2-4573-4250-929e-0cd7d12e2070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2e8765b-gb75g\" (UID: \"994b0fe2-4573-4250-929e-0cd7d12e2070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"
Nov 02 09:49:14.817373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.814964    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xwj\" (UniqueName: \"kubernetes.io/projected/994b0fe2-4573-4250-929e-0cd7d12e2070-kube-api-access-r5xwj\") pod \"maas-api-revoke-sub-c2e8765b-gb75g\" (UID: \"994b0fe2-4573-4250-929e-0cd7d12e2070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"
Nov 02 09:49:14.893240 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:14.893161    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"
Nov 02 09:49:15.178398 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:15.178365    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"]
Nov 02 09:49:15.179733 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:49:15.179709    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994b0fe2_4573_4250_929e_0cd7d12e2070.slice/crio-276a0fade7bcd7c25e18429633b1a41268ac92f6c5a07644604a5f9daaf4b58c WatchSource:0}: Error finding container 276a0fade7bcd7c25e18429633b1a41268ac92f6c5a07644604a5f9daaf4b58c: Status 404 returned error can't find the container with id 276a0fade7bcd7c25e18429633b1a41268ac92f6c5a07644604a5f9daaf4b58c
Nov 02 09:49:15.264560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:15.264526    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g" event={"ID":"994b0fe2-4573-4250-929e-0cd7d12e2070","Type":"ContainerStarted","Data":"276a0fade7bcd7c25e18429633b1a41268ac92f6c5a07644604a5f9daaf4b58c"}
Nov 02 09:49:16.271605 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:16.271570    4614 generic.go:356] "Generic (PLEG): container finished" podID="994b0fe2-4573-4250-929e-0cd7d12e2070" containerID="ede096370a01f03fb23abd46338ca2e57f528f282f6ee3190730778bf6189eb2" exitCode=0
Nov 02 09:49:16.272028 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:16.271621    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g" event={"ID":"994b0fe2-4573-4250-929e-0cd7d12e2070","Type":"ContainerDied","Data":"ede096370a01f03fb23abd46338ca2e57f528f282f6ee3190730778bf6189eb2"}
Nov 02 09:49:17.435054 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:17.435024    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"
Nov 02 09:49:17.526580 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:17.526544    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994b0fe2-4573-4250-929e-0cd7d12e2070-kube-api-access-r5xwj\" (UniqueName: \"kubernetes.io/projected/994b0fe2-4573-4250-929e-0cd7d12e2070-kube-api-access-r5xwj\") pod \"994b0fe2-4573-4250-929e-0cd7d12e2070\" (UID: \"994b0fe2-4573-4250-929e-0cd7d12e2070\") "
Nov 02 09:49:17.526771 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:17.526588    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994b0fe2-4573-4250-929e-0cd7d12e2070-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994b0fe2-4573-4250-929e-0cd7d12e2070-maas-api-openshift-ca\") pod \"994b0fe2-4573-4250-929e-0cd7d12e2070\" (UID: \"994b0fe2-4573-4250-929e-0cd7d12e2070\") "
Nov 02 09:49:17.526831 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:17.526790    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994b0fe2-4573-4250-929e-0cd7d12e2070-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994b0fe2-4573-4250-929e-0cd7d12e2070-maas-api-xks-ca\") pod \"994b0fe2-4573-4250-929e-0cd7d12e2070\" (UID: \"994b0fe2-4573-4250-929e-0cd7d12e2070\") "
Nov 02 09:49:17.527014 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:17.526989    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994b0fe2-4573-4250-929e-0cd7d12e2070-maas-api-openshift-ca" pod "994b0fe2-4573-4250-929e-0cd7d12e2070" (UID: "994b0fe2-4573-4250-929e-0cd7d12e2070"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:17.527238 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:17.527221    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994b0fe2-4573-4250-929e-0cd7d12e2070-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:17.528847 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:17.528790    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994b0fe2-4573-4250-929e-0cd7d12e2070-maas-api-xks-ca" pod "994b0fe2-4573-4250-929e-0cd7d12e2070" (UID: "994b0fe2-4573-4250-929e-0cd7d12e2070"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:49:17.528944 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:17.528884    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994b0fe2-4573-4250-929e-0cd7d12e2070-kube-api-access-r5xwj" pod "994b0fe2-4573-4250-929e-0cd7d12e2070" (UID: "994b0fe2-4573-4250-929e-0cd7d12e2070"). InnerVolumeSpecName "kube-api-access-r5xwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:17.628688 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:17.628644    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994b0fe2-4573-4250-929e-0cd7d12e2070-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:17.628688 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:17.628682    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5xwj\" (UniqueName: \"kubernetes.io/projected/994b0fe2-4573-4250-929e-0cd7d12e2070-kube-api-access-r5xwj\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:18.383850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:18.383811    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g" event={"ID":"994b0fe2-4573-4250-929e-0cd7d12e2070","Type":"ContainerDied","Data":"276a0fade7bcd7c25e18429633b1a41268ac92f6c5a07644604a5f9daaf4b58c"}
Nov 02 09:49:18.384118 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:18.384099    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="276a0fade7bcd7c25e18429633b1a41268ac92f6c5a07644604a5f9daaf4b58c"
Nov 02 09:49:18.384304 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:18.384290    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"
Nov 02 09:49:22.065379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:22.064495    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:22.630960 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:22.630916    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:23.627574 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:23.627538    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"]
Nov 02 09:49:25.839912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:25.839870    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"]
Nov 02 09:49:25.840495 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:25.840478    4614 cpu_manager.go:414] "containerMap: removing container" podUID="994b0fe2-4573-4250-929e-0cd7d12e2070" containerName="revoke-keys"
Nov 02 09:49:25.840557 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:25.840496    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994b0fe2-4573-4250-929e-0cd7d12e2070" containerName="revoke-keys"
Nov 02 09:49:25.840627 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:25.840616    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="994b0fe2-4573-4250-929e-0cd7d12e2070" containerName="revoke-keys"
Nov 02 09:49:25.845316 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:25.845293    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"
Nov 02 09:49:25.855287 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:25.855237    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:49:25.855474 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:25.855317    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:49:25.857498 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:25.857470    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"]
Nov 02 09:49:25.923634 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:25.923610    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0311b3-9a35-49a4-81db-2cccfc296cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cfeb2c2-pjxfq\" (UID: \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"
Nov 02 09:49:25.923753 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:25.923651    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0311b3-9a35-49a4-81db-2cccfc296cf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cfeb2c2-pjxfq\" (UID: \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"
Nov 02 09:49:25.923880 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:25.923857    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9dwf\" (UniqueName: \"kubernetes.io/projected/ae0311b3-9a35-49a4-81db-2cccfc296cf2-kube-api-access-r9dwf\") pod \"maas-api-revoke-sub-5cfeb2c2-pjxfq\" (UID: \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"
Nov 02 09:49:26.025418 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:26.025386    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9dwf\" (UniqueName: \"kubernetes.io/projected/ae0311b3-9a35-49a4-81db-2cccfc296cf2-kube-api-access-r9dwf\") pod \"maas-api-revoke-sub-5cfeb2c2-pjxfq\" (UID: \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"
Nov 02 09:49:26.025597 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:26.025502    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0311b3-9a35-49a4-81db-2cccfc296cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cfeb2c2-pjxfq\" (UID: \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"
Nov 02 09:49:26.025597 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:26.025544    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0311b3-9a35-49a4-81db-2cccfc296cf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cfeb2c2-pjxfq\" (UID: \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"
Nov 02 09:49:26.026186 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:26.026163    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0311b3-9a35-49a4-81db-2cccfc296cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cfeb2c2-pjxfq\" (UID: \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"
Nov 02 09:49:26.027902 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:26.027882    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0311b3-9a35-49a4-81db-2cccfc296cf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cfeb2c2-pjxfq\" (UID: \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"
Nov 02 09:49:26.041616 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:26.041579    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9dwf\" (UniqueName: \"kubernetes.io/projected/ae0311b3-9a35-49a4-81db-2cccfc296cf2-kube-api-access-r9dwf\") pod \"maas-api-revoke-sub-5cfeb2c2-pjxfq\" (UID: \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"
Nov 02 09:49:26.184590 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:26.184512    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"
Nov 02 09:49:26.485312 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:26.485286    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"]
Nov 02 09:49:26.487661 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:49:26.487628    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0311b3_9a35_49a4_81db_2cccfc296cf2.slice/crio-1c0dc607b961ed7fd5e39505346b29910596d8c3afaf88c0c0ad01a9ef30e32e WatchSource:0}: Error finding container 1c0dc607b961ed7fd5e39505346b29910596d8c3afaf88c0c0ad01a9ef30e32e: Status 404 returned error can't find the container with id 1c0dc607b961ed7fd5e39505346b29910596d8c3afaf88c0c0ad01a9ef30e32e
Nov 02 09:49:26.747541 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:26.747506    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq" event={"ID":"ae0311b3-9a35-49a4-81db-2cccfc296cf2","Type":"ContainerStarted","Data":"6f952144c45b318ef83ea3d976a5d2230cddb77ff4b81a9a19510cfd4479a6b3"}
Nov 02 09:49:26.747653 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:26.747568    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq" event={"ID":"ae0311b3-9a35-49a4-81db-2cccfc296cf2","Type":"ContainerStarted","Data":"1c0dc607b961ed7fd5e39505346b29910596d8c3afaf88c0c0ad01a9ef30e32e"}
Nov 02 09:49:26.766596 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:26.766553    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq" podStartSLOduration=1.766541089 podStartE2EDuration="1.766541089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:49:26.765298361 +0000 UTC m=+847.597956442" watchObservedRunningTime="2026-11-02 09:49:26.766541089 +0000 UTC m=+847.599199169"
Nov 02 09:49:27.755137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:27.755103    4614 generic.go:356] "Generic (PLEG): container finished" podID="ae0311b3-9a35-49a4-81db-2cccfc296cf2" containerID="6f952144c45b318ef83ea3d976a5d2230cddb77ff4b81a9a19510cfd4479a6b3" exitCode=0
Nov 02 09:49:27.755585 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:27.755168    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq" event={"ID":"ae0311b3-9a35-49a4-81db-2cccfc296cf2","Type":"ContainerDied","Data":"6f952144c45b318ef83ea3d976a5d2230cddb77ff4b81a9a19510cfd4479a6b3"}
Nov 02 09:49:28.628723 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.628684    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv" podUID="86364d67-8d18-4d74-94ee-b03b7bbe399f" containerName="payload-processing" containerID="cri-o://9849fe3addb3deeeeba7793e8c43267ce378f0c5ef7db067ee05a6543bc74a61" gracePeriod=25
Nov 02 09:49:28.780972 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.780526    4614 generic.go:356] "Generic (PLEG): container finished" podID="86364d67-8d18-4d74-94ee-b03b7bbe399f" containerID="9849fe3addb3deeeeba7793e8c43267ce378f0c5ef7db067ee05a6543bc74a61" exitCode=0
Nov 02 09:49:28.780972 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.780803    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv" event={"ID":"86364d67-8d18-4d74-94ee-b03b7bbe399f","Type":"ContainerDied","Data":"9849fe3addb3deeeeba7793e8c43267ce378f0c5ef7db067ee05a6543bc74a61"}
Nov 02 09:49:28.951852 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.951819    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:49:28.989259 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.989227    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86364d67-8d18-4d74-94ee-b03b7bbe399f-kube-api-access-jg8gj\" (UniqueName: \"kubernetes.io/projected/86364d67-8d18-4d74-94ee-b03b7bbe399f-kube-api-access-jg8gj\") pod \"86364d67-8d18-4d74-94ee-b03b7bbe399f\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") "
Nov 02 09:49:28.989421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.989276    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86364d67-8d18-4d74-94ee-b03b7bbe399f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86364d67-8d18-4d74-94ee-b03b7bbe399f-plugins-config-volume\") pod \"86364d67-8d18-4d74-94ee-b03b7bbe399f\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") "
Nov 02 09:49:28.989489 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.989451    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86364d67-8d18-4d74-94ee-b03b7bbe399f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/86364d67-8d18-4d74-94ee-b03b7bbe399f-provider-credentials\") pod \"86364d67-8d18-4d74-94ee-b03b7bbe399f\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") "
Nov 02 09:49:28.989552 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.989492    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86364d67-8d18-4d74-94ee-b03b7bbe399f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86364d67-8d18-4d74-94ee-b03b7bbe399f-routing-overlay-volume\") pod \"86364d67-8d18-4d74-94ee-b03b7bbe399f\" (UID: \"86364d67-8d18-4d74-94ee-b03b7bbe399f\") "
Nov 02 09:49:28.989980 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.989939    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86364d67-8d18-4d74-94ee-b03b7bbe399f-routing-overlay-volume" pod "86364d67-8d18-4d74-94ee-b03b7bbe399f" (UID: "86364d67-8d18-4d74-94ee-b03b7bbe399f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:28.990095 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.990033    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86364d67-8d18-4d74-94ee-b03b7bbe399f-plugins-config-volume" pod "86364d67-8d18-4d74-94ee-b03b7bbe399f" (UID: "86364d67-8d18-4d74-94ee-b03b7bbe399f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:28.990688 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.990659    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86364d67-8d18-4d74-94ee-b03b7bbe399f-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:28.990792 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.990692    4614 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86364d67-8d18-4d74-94ee-b03b7bbe399f-routing-overlay-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:28.991867 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.991840    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86364d67-8d18-4d74-94ee-b03b7bbe399f-kube-api-access-jg8gj" pod "86364d67-8d18-4d74-94ee-b03b7bbe399f" (UID: "86364d67-8d18-4d74-94ee-b03b7bbe399f"). InnerVolumeSpecName "kube-api-access-jg8gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:28.992029 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.992002    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86364d67-8d18-4d74-94ee-b03b7bbe399f-provider-credentials" pod "86364d67-8d18-4d74-94ee-b03b7bbe399f" (UID: "86364d67-8d18-4d74-94ee-b03b7bbe399f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:28.999069 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:28.999046    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"
Nov 02 09:49:29.091928 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.091895    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae0311b3-9a35-49a4-81db-2cccfc296cf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0311b3-9a35-49a4-81db-2cccfc296cf2-maas-api-openshift-ca\") pod \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\" (UID: \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\") "
Nov 02 09:49:29.092133 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.091979    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae0311b3-9a35-49a4-81db-2cccfc296cf2-kube-api-access-r9dwf\" (UniqueName: \"kubernetes.io/projected/ae0311b3-9a35-49a4-81db-2cccfc296cf2-kube-api-access-r9dwf\") pod \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\" (UID: \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\") "
Nov 02 09:49:29.092133 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.092103    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae0311b3-9a35-49a4-81db-2cccfc296cf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0311b3-9a35-49a4-81db-2cccfc296cf2-maas-api-xks-ca\") pod \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\" (UID: \"ae0311b3-9a35-49a4-81db-2cccfc296cf2\") "
Nov 02 09:49:29.092309 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.092281    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae0311b3-9a35-49a4-81db-2cccfc296cf2-maas-api-openshift-ca" pod "ae0311b3-9a35-49a4-81db-2cccfc296cf2" (UID: "ae0311b3-9a35-49a4-81db-2cccfc296cf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:29.092569 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.092550    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg8gj\" (UniqueName: \"kubernetes.io/projected/86364d67-8d18-4d74-94ee-b03b7bbe399f-kube-api-access-jg8gj\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:29.092621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.092572    4614 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86364d67-8d18-4d74-94ee-b03b7bbe399f-provider-credentials\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:29.092621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.092582    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0311b3-9a35-49a4-81db-2cccfc296cf2-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:29.094165 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.094140    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0311b3-9a35-49a4-81db-2cccfc296cf2-kube-api-access-r9dwf" pod "ae0311b3-9a35-49a4-81db-2cccfc296cf2" (UID: "ae0311b3-9a35-49a4-81db-2cccfc296cf2"). InnerVolumeSpecName "kube-api-access-r9dwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:29.094240 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.094174    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae0311b3-9a35-49a4-81db-2cccfc296cf2-maas-api-xks-ca" pod "ae0311b3-9a35-49a4-81db-2cccfc296cf2" (UID: "ae0311b3-9a35-49a4-81db-2cccfc296cf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:49:29.194009 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.193916    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9dwf\" (UniqueName: \"kubernetes.io/projected/ae0311b3-9a35-49a4-81db-2cccfc296cf2-kube-api-access-r9dwf\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:29.194009 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.193945    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0311b3-9a35-49a4-81db-2cccfc296cf2-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:29.788116 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.788090    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"
Nov 02 09:49:29.788589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.788140    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv" event={"ID":"86364d67-8d18-4d74-94ee-b03b7bbe399f","Type":"ContainerDied","Data":"d0053486f1cf457efed2b9947d992d35a4edd694eceeb27632d6fa6d1f5fb8a8"}
Nov 02 09:49:29.788589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.788212    4614 scope.go:122] "RemoveContainer" containerID="9849fe3addb3deeeeba7793e8c43267ce378f0c5ef7db067ee05a6543bc74a61"
Nov 02 09:49:29.789800 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.789767    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq" event={"ID":"ae0311b3-9a35-49a4-81db-2cccfc296cf2","Type":"ContainerDied","Data":"1c0dc607b961ed7fd5e39505346b29910596d8c3afaf88c0c0ad01a9ef30e32e"}
Nov 02 09:49:29.789800 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.789793    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"
Nov 02 09:49:29.789946 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.789822    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c0dc607b961ed7fd5e39505346b29910596d8c3afaf88c0c0ad01a9ef30e32e"
Nov 02 09:49:29.819252 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.819220    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"]
Nov 02 09:49:29.819927 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.819884    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-7898vhvpv"]
Nov 02 09:49:29.967068 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.966989    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"]
Nov 02 09:49:29.967542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.967527    4614 cpu_manager.go:414] "containerMap: removing container" podUID="86364d67-8d18-4d74-94ee-b03b7bbe399f" containerName="payload-processing"
Nov 02 09:49:29.967621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.967543    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86364d67-8d18-4d74-94ee-b03b7bbe399f" containerName="payload-processing"
Nov 02 09:49:29.967621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.967552    4614 cpu_manager.go:414] "containerMap: removing container" podUID="ae0311b3-9a35-49a4-81db-2cccfc296cf2" containerName="revoke-keys"
Nov 02 09:49:29.967621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.967557    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae0311b3-9a35-49a4-81db-2cccfc296cf2" containerName="revoke-keys"
Nov 02 09:49:29.967714 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.967638    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="86364d67-8d18-4d74-94ee-b03b7bbe399f" containerName="payload-processing"
Nov 02 09:49:29.967714 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.967645    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae0311b3-9a35-49a4-81db-2cccfc296cf2" containerName="revoke-keys"
Nov 02 09:49:29.972216 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.972196    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"
Nov 02 09:49:29.976169 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.976146    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:49:29.976594 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.976413    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:49:29.980496 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:29.980467    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"]
Nov 02 09:49:30.002319 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:30.002238    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dff8b74-45bc-40e3-8b16-4a8010a6e162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17214e47-zpzt8\" (UID: \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"
Nov 02 09:49:30.002319 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:30.002275    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bbnc\" (UniqueName: \"kubernetes.io/projected/0dff8b74-45bc-40e3-8b16-4a8010a6e162-kube-api-access-4bbnc\") pod \"maas-api-revoke-sub-17214e47-zpzt8\" (UID: \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"
Nov 02 09:49:30.002552 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:30.002433    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dff8b74-45bc-40e3-8b16-4a8010a6e162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17214e47-zpzt8\" (UID: \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"
Nov 02 09:49:30.153413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:30.107121    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dff8b74-45bc-40e3-8b16-4a8010a6e162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17214e47-zpzt8\" (UID: \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"
Nov 02 09:49:30.168396 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:30.153796    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bbnc\" (UniqueName: \"kubernetes.io/projected/0dff8b74-45bc-40e3-8b16-4a8010a6e162-kube-api-access-4bbnc\") pod \"maas-api-revoke-sub-17214e47-zpzt8\" (UID: \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"
Nov 02 09:49:30.168729 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:30.168704    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dff8b74-45bc-40e3-8b16-4a8010a6e162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17214e47-zpzt8\" (UID: \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"
Nov 02 09:49:30.170602 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:30.169541    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dff8b74-45bc-40e3-8b16-4a8010a6e162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17214e47-zpzt8\" (UID: \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"
Nov 02 09:49:30.173617 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:30.173580    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bbnc\" (UniqueName: \"kubernetes.io/projected/0dff8b74-45bc-40e3-8b16-4a8010a6e162-kube-api-access-4bbnc\") pod \"maas-api-revoke-sub-17214e47-zpzt8\" (UID: \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"
Nov 02 09:49:30.174007 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:30.173982    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dff8b74-45bc-40e3-8b16-4a8010a6e162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17214e47-zpzt8\" (UID: \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"
Nov 02 09:49:30.288548 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:30.288519    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"
Nov 02 09:49:30.566827 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:30.566796    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"]
Nov 02 09:49:30.567104 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:49:30.567081    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dff8b74_45bc_40e3_8b16_4a8010a6e162.slice/crio-dfb5d57ee3bf354e1d0d9b5799eff8ad32f4dba5b6079e097dd46e556553ddd8 WatchSource:0}: Error finding container dfb5d57ee3bf354e1d0d9b5799eff8ad32f4dba5b6079e097dd46e556553ddd8: Status 404 returned error can't find the container with id dfb5d57ee3bf354e1d0d9b5799eff8ad32f4dba5b6079e097dd46e556553ddd8
Nov 02 09:49:30.797515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:30.797483    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8" event={"ID":"0dff8b74-45bc-40e3-8b16-4a8010a6e162","Type":"ContainerStarted","Data":"dfb5d57ee3bf354e1d0d9b5799eff8ad32f4dba5b6079e097dd46e556553ddd8"}
Nov 02 09:49:31.316443 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:31.316406    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86364d67-8d18-4d74-94ee-b03b7bbe399f" path="/var/lib/kubelet/pods/86364d67-8d18-4d74-94ee-b03b7bbe399f/volumes"
Nov 02 09:49:31.805579 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:31.805547    4614 generic.go:356] "Generic (PLEG): container finished" podID="0dff8b74-45bc-40e3-8b16-4a8010a6e162" containerID="d5938b107644a6149c9e517c9e6237a5882daf90858d4aeee0992bec09de49b7" exitCode=0
Nov 02 09:49:31.805976 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:31.805636    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8" event={"ID":"0dff8b74-45bc-40e3-8b16-4a8010a6e162","Type":"ContainerDied","Data":"d5938b107644a6149c9e517c9e6237a5882daf90858d4aeee0992bec09de49b7"}
Nov 02 09:49:34.000384 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:33.998088    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"
Nov 02 09:49:34.053376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.048736    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dff8b74-45bc-40e3-8b16-4a8010a6e162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dff8b74-45bc-40e3-8b16-4a8010a6e162-maas-api-xks-ca\") pod \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\" (UID: \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\") "
Nov 02 09:49:34.053376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.048815    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dff8b74-45bc-40e3-8b16-4a8010a6e162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dff8b74-45bc-40e3-8b16-4a8010a6e162-maas-api-openshift-ca\") pod \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\" (UID: \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\") "
Nov 02 09:49:34.053376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.048960    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dff8b74-45bc-40e3-8b16-4a8010a6e162-kube-api-access-4bbnc\" (UniqueName: \"kubernetes.io/projected/0dff8b74-45bc-40e3-8b16-4a8010a6e162-kube-api-access-4bbnc\") pod \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\" (UID: \"0dff8b74-45bc-40e3-8b16-4a8010a6e162\") "
Nov 02 09:49:34.067376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.064750    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dff8b74-45bc-40e3-8b16-4a8010a6e162-maas-api-openshift-ca" pod "0dff8b74-45bc-40e3-8b16-4a8010a6e162" (UID: "0dff8b74-45bc-40e3-8b16-4a8010a6e162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:34.067376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.064855    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:34.086994 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.086951    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dff8b74-45bc-40e3-8b16-4a8010a6e162-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:34.122810 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.122758    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:34.210476 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.183669    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:34.381825 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.381772    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dff8b74-45bc-40e3-8b16-4a8010a6e162-kube-api-access-4bbnc" pod "0dff8b74-45bc-40e3-8b16-4a8010a6e162" (UID: "0dff8b74-45bc-40e3-8b16-4a8010a6e162"). InnerVolumeSpecName "kube-api-access-4bbnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:34.400398 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.391406    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bbnc\" (UniqueName: \"kubernetes.io/projected/0dff8b74-45bc-40e3-8b16-4a8010a6e162-kube-api-access-4bbnc\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:34.467425 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.456452    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 09:49:34.467425 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.461492    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dff8b74-45bc-40e3-8b16-4a8010a6e162-maas-api-xks-ca" pod "0dff8b74-45bc-40e3-8b16-4a8010a6e162" (UID: "0dff8b74-45bc-40e3-8b16-4a8010a6e162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:49:34.505292 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.505250    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dff8b74-45bc-40e3-8b16-4a8010a6e162-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:34.642376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.631497    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 02 09:49:34.951377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.950726    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8" event={"ID":"0dff8b74-45bc-40e3-8b16-4a8010a6e162","Type":"ContainerDied","Data":"dfb5d57ee3bf354e1d0d9b5799eff8ad32f4dba5b6079e097dd46e556553ddd8"}
Nov 02 09:49:34.951377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.950790    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfb5d57ee3bf354e1d0d9b5799eff8ad32f4dba5b6079e097dd46e556553ddd8"
Nov 02 09:49:34.951377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:34.950855    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"
Nov 02 09:49:36.179865 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.179832    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"]
Nov 02 09:49:36.180403 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.180373    4614 cpu_manager.go:414] "containerMap: removing container" podUID="0dff8b74-45bc-40e3-8b16-4a8010a6e162" containerName="revoke-keys"
Nov 02 09:49:36.180403 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.180388    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dff8b74-45bc-40e3-8b16-4a8010a6e162" containerName="revoke-keys"
Nov 02 09:49:36.180506 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.180463    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dff8b74-45bc-40e3-8b16-4a8010a6e162" containerName="revoke-keys"
Nov 02 09:49:36.183990 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.183970    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"
Nov 02 09:49:36.192010 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.191524    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:49:36.192010 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.191579    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:49:36.195504 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.195482    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"]
Nov 02 09:49:36.296865 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.296827    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d67cd3d-26dd-4e86-9445-f011a3be1666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-950b3b12-k8rwn\" (UID: \"1d67cd3d-26dd-4e86-9445-f011a3be1666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"
Nov 02 09:49:36.297065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.297048    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d67cd3d-26dd-4e86-9445-f011a3be1666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-950b3b12-k8rwn\" (UID: \"1d67cd3d-26dd-4e86-9445-f011a3be1666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"
Nov 02 09:49:36.297125 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.297098    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbj9p\" (UniqueName: \"kubernetes.io/projected/1d67cd3d-26dd-4e86-9445-f011a3be1666-kube-api-access-sbj9p\") pod \"maas-api-revoke-sub-950b3b12-k8rwn\" (UID: \"1d67cd3d-26dd-4e86-9445-f011a3be1666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"
Nov 02 09:49:36.398078 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.398040    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbj9p\" (UniqueName: \"kubernetes.io/projected/1d67cd3d-26dd-4e86-9445-f011a3be1666-kube-api-access-sbj9p\") pod \"maas-api-revoke-sub-950b3b12-k8rwn\" (UID: \"1d67cd3d-26dd-4e86-9445-f011a3be1666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"
Nov 02 09:49:36.398251 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.398122    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d67cd3d-26dd-4e86-9445-f011a3be1666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-950b3b12-k8rwn\" (UID: \"1d67cd3d-26dd-4e86-9445-f011a3be1666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"
Nov 02 09:49:36.398307 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.398289    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d67cd3d-26dd-4e86-9445-f011a3be1666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-950b3b12-k8rwn\" (UID: \"1d67cd3d-26dd-4e86-9445-f011a3be1666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"
Nov 02 09:49:36.398961 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.398939    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d67cd3d-26dd-4e86-9445-f011a3be1666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-950b3b12-k8rwn\" (UID: \"1d67cd3d-26dd-4e86-9445-f011a3be1666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"
Nov 02 09:49:36.400565 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.400543    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d67cd3d-26dd-4e86-9445-f011a3be1666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-950b3b12-k8rwn\" (UID: \"1d67cd3d-26dd-4e86-9445-f011a3be1666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"
Nov 02 09:49:36.414642 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.414616    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbj9p\" (UniqueName: \"kubernetes.io/projected/1d67cd3d-26dd-4e86-9445-f011a3be1666-kube-api-access-sbj9p\") pod \"maas-api-revoke-sub-950b3b12-k8rwn\" (UID: \"1d67cd3d-26dd-4e86-9445-f011a3be1666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"
Nov 02 09:49:36.501617 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.501577    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"
Nov 02 09:49:36.783390 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.783361    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"]
Nov 02 09:49:36.784514 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:49:36.784487    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d67cd3d_26dd_4e86_9445_f011a3be1666.slice/crio-90b3e62a670fef9e05ca53db9be3d42a0a4444e3bc78924aab29813dd8ceab33 WatchSource:0}: Error finding container 90b3e62a670fef9e05ca53db9be3d42a0a4444e3bc78924aab29813dd8ceab33: Status 404 returned error can't find the container with id 90b3e62a670fef9e05ca53db9be3d42a0a4444e3bc78924aab29813dd8ceab33
Nov 02 09:49:36.973801 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:36.973767    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn" event={"ID":"1d67cd3d-26dd-4e86-9445-f011a3be1666","Type":"ContainerStarted","Data":"90b3e62a670fef9e05ca53db9be3d42a0a4444e3bc78924aab29813dd8ceab33"}
Nov 02 09:49:37.489104 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.489065    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"]
Nov 02 09:49:37.492975 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.492955    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.497179 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.497147    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-bde944-openshift-default-dockercfg-hb9wq\""
Nov 02 09:49:37.507050 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.507023    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"]
Nov 02 09:49:37.612877 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.612836    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.613046 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.612880    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.613046 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.612992    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1137058f-bb14-42e6-ac55-d2104536ffcc-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.613046 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.613024    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.613168 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.613055    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.613211 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.613160    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.613211 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.613186    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4w9m\" (UniqueName: \"kubernetes.io/projected/1137058f-bb14-42e6-ac55-d2104536ffcc-kube-api-access-x4w9m\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.613309 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.613288    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.613346 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.613336    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.714487 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.714451    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.714647 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.714578    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.714647 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.714607    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.714793 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.714676    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1137058f-bb14-42e6-ac55-d2104536ffcc-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.714793 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.714704    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.714793 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.714749    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.714793 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.714780    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.714984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.714803    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4w9m\" (UniqueName: \"kubernetes.io/projected/1137058f-bb14-42e6-ac55-d2104536ffcc-kube-api-access-x4w9m\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.714984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.714875    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.715124 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.715087    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.715194 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.715168    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.715250 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.715214    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.715324 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.715304    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.715474 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.715457    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1137058f-bb14-42e6-ac55-d2104536ffcc-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.716937 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.716913    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.717161 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.717142    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.729219 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.729177    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.729375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.729229    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4w9m\" (UniqueName: \"kubernetes.io/projected/1137058f-bb14-42e6-ac55-d2104536ffcc-kube-api-access-x4w9m\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.825566 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.825222    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:37.983066 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.983032    4614 generic.go:356] "Generic (PLEG): container finished" podID="1d67cd3d-26dd-4e86-9445-f011a3be1666" containerID="019fcd0a589e89f51219810d9f03686919fd8973b1a7b02e60ac0d3493ac0539" exitCode=0
Nov 02 09:49:37.983217 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:37.983120    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn" event={"ID":"1d67cd3d-26dd-4e86-9445-f011a3be1666","Type":"ContainerDied","Data":"019fcd0a589e89f51219810d9f03686919fd8973b1a7b02e60ac0d3493ac0539"}
Nov 02 09:49:38.142250 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:49:38.142208    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1137058f_bb14_42e6_ac55_d2104536ffcc.slice/crio-b8743c6be3eb9e54123dd15b95a1b9be2aa1c6efa58b5a3d90694ff92ab7bd2d WatchSource:0}: Error finding container b8743c6be3eb9e54123dd15b95a1b9be2aa1c6efa58b5a3d90694ff92ab7bd2d: Status 404 returned error can't find the container with id b8743c6be3eb9e54123dd15b95a1b9be2aa1c6efa58b5a3d90694ff92ab7bd2d
Nov 02 09:49:38.142505 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:38.142486    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"]
Nov 02 09:49:38.145822 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:38.145782    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:49:38.145911 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:38.145843    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:49:38.145911 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:38.145869    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:49:38.991152 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:38.991113    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62" event={"ID":"1137058f-bb14-42e6-ac55-d2104536ffcc","Type":"ContainerStarted","Data":"9df7346abfbbe42e93126985c74796bc917a0b91aa0faacf3823be70b18d742a"}
Nov 02 09:49:38.991586 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:38.991180    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62" event={"ID":"1137058f-bb14-42e6-ac55-d2104536ffcc","Type":"ContainerStarted","Data":"b8743c6be3eb9e54123dd15b95a1b9be2aa1c6efa58b5a3d90694ff92ab7bd2d"}
Nov 02 09:49:39.019484 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.019434    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62" podStartSLOduration=2.019419424 podStartE2EDuration="2.019419424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:49:39.017522424 +0000 UTC m=+859.850180503" watchObservedRunningTime="2026-11-02 09:49:39.019419424 +0000 UTC m=+859.852077504"
Nov 02 09:49:39.200593 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.200561    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"
Nov 02 09:49:39.368117 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.368021    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"]
Nov 02 09:49:39.369158 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.368811    4614 cpu_manager.go:414] "containerMap: removing container" podUID="1d67cd3d-26dd-4e86-9445-f011a3be1666" containerName="revoke-keys"
Nov 02 09:49:39.369158 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.368845    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d67cd3d-26dd-4e86-9445-f011a3be1666" containerName="revoke-keys"
Nov 02 09:49:39.369158 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.369023    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d67cd3d-26dd-4e86-9445-f011a3be1666" containerName="revoke-keys"
Nov 02 09:49:39.371118 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.370951    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d67cd3d-26dd-4e86-9445-f011a3be1666-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d67cd3d-26dd-4e86-9445-f011a3be1666-maas-api-openshift-ca\") pod \"1d67cd3d-26dd-4e86-9445-f011a3be1666\" (UID: \"1d67cd3d-26dd-4e86-9445-f011a3be1666\") "
Nov 02 09:49:39.371118 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.371035    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d67cd3d-26dd-4e86-9445-f011a3be1666-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d67cd3d-26dd-4e86-9445-f011a3be1666-maas-api-xks-ca\") pod \"1d67cd3d-26dd-4e86-9445-f011a3be1666\" (UID: \"1d67cd3d-26dd-4e86-9445-f011a3be1666\") "
Nov 02 09:49:39.371118 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.371099    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d67cd3d-26dd-4e86-9445-f011a3be1666-kube-api-access-sbj9p\" (UniqueName: \"kubernetes.io/projected/1d67cd3d-26dd-4e86-9445-f011a3be1666-kube-api-access-sbj9p\") pod \"1d67cd3d-26dd-4e86-9445-f011a3be1666\" (UID: \"1d67cd3d-26dd-4e86-9445-f011a3be1666\") "
Nov 02 09:49:39.371829 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.371805    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d67cd3d-26dd-4e86-9445-f011a3be1666-maas-api-openshift-ca" pod "1d67cd3d-26dd-4e86-9445-f011a3be1666" (UID: "1d67cd3d-26dd-4e86-9445-f011a3be1666"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:49:39.373167 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.373140    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.375270 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.375244    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"]
Nov 02 09:49:39.378596 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.378557    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2e80fb\"/\"distinct-w0-2e80fb-kserve-self-signed-certs\""
Nov 02 09:49:39.382782 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.382748    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d67cd3d-26dd-4e86-9445-f011a3be1666-maas-api-xks-ca" pod "1d67cd3d-26dd-4e86-9445-f011a3be1666" (UID: "1d67cd3d-26dd-4e86-9445-f011a3be1666"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:49:39.387366 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.387312    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d67cd3d-26dd-4e86-9445-f011a3be1666-kube-api-access-sbj9p" pod "1d67cd3d-26dd-4e86-9445-f011a3be1666" (UID: "1d67cd3d-26dd-4e86-9445-f011a3be1666"). InnerVolumeSpecName "kube-api-access-sbj9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:49:39.472492 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.472450    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvsf\" (UniqueName: \"kubernetes.io/projected/19dca143-7178-44e5-9e3e-593f99e267c9-kube-api-access-vjvsf\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.472703 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.472581    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-model-cache\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.472703 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.472633    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-home\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.472831 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.472741    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-dshm\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.472831 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.472771    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-tmp-dir\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.472831 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.472812    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19dca143-7178-44e5-9e3e-593f99e267c9-tls-certs\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.473005 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.472987    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d67cd3d-26dd-4e86-9445-f011a3be1666-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:39.473061 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.473008    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d67cd3d-26dd-4e86-9445-f011a3be1666-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:39.473061 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.473019    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbj9p\" (UniqueName: \"kubernetes.io/projected/1d67cd3d-26dd-4e86-9445-f011a3be1666-kube-api-access-sbj9p\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:49:39.582412 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.582366    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-model-cache\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.582674 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.582651    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-home\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.582859 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.582844    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-dshm\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.582971 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.582957    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-tmp-dir\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.583086 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.583073    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19dca143-7178-44e5-9e3e-593f99e267c9-tls-certs\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.583288 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.583273    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvsf\" (UniqueName: \"kubernetes.io/projected/19dca143-7178-44e5-9e3e-593f99e267c9-kube-api-access-vjvsf\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.584146 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.584118    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-model-cache\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.584621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.584599    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-home\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.585376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.585300    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-tmp-dir\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.607655 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.607623    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-dshm\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.608440 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.608414    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19dca143-7178-44e5-9e3e-593f99e267c9-tls-certs\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.614167 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.614112    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvsf\" (UniqueName: \"kubernetes.io/projected/19dca143-7178-44e5-9e3e-593f99e267c9-kube-api-access-vjvsf\") pod \"distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") " pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.715583 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.715212    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:39.752340 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:39.752307    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:49:39.862287 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.859593    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:39.862287 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.859823    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 09:49:39.862287 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:39.859869    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62" podUID="1137058f-bb14-42e6-ac55-d2104536ffcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 09:49:40.006222 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:40.006173    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn" event={"ID":"1d67cd3d-26dd-4e86-9445-f011a3be1666","Type":"ContainerDied","Data":"90b3e62a670fef9e05ca53db9be3d42a0a4444e3bc78924aab29813dd8ceab33"}
Nov 02 09:49:40.006726 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:40.006187    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"
Nov 02 09:49:40.006726 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:40.006278    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90b3e62a670fef9e05ca53db9be3d42a0a4444e3bc78924aab29813dd8ceab33"
Nov 02 09:49:40.055892 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:40.055225    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"]
Nov 02 09:49:40.058402 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:49:40.058321    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19dca143_7178_44e5_9e3e_593f99e267c9.slice/crio-b649e428b6011ce5002e848d53ecd52f6cc50651381fa33fc4226d2792c18028 WatchSource:0}: Error finding container b649e428b6011ce5002e848d53ecd52f6cc50651381fa33fc4226d2792c18028: Status 404 returned error can't find the container with id b649e428b6011ce5002e848d53ecd52f6cc50651381fa33fc4226d2792c18028
Nov 02 09:49:40.829787 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:40.829750    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 09:49:40.830041 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:40.830014    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62" podUID="1137058f-bb14-42e6-ac55-d2104536ffcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 09:49:41.014650 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:41.014614    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" event={"ID":"19dca143-7178-44e5-9e3e-593f99e267c9","Type":"ContainerStarted","Data":"b6442cc09e0d7703e2562def86e30c153d5ba0929e7771f55504d11cdffaa379"}
Nov 02 09:49:41.015017 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:41.014663    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" event={"ID":"19dca143-7178-44e5-9e3e-593f99e267c9","Type":"ContainerStarted","Data":"b649e428b6011ce5002e848d53ecd52f6cc50651381fa33fc4226d2792c18028"}
Nov 02 09:49:41.716760 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:41.716722    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:41.829377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:41.826971    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 09:49:41.829377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:41.827030    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62" podUID="1137058f-bb14-42e6-ac55-d2104536ffcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 09:49:42.829378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:42.828724    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 09:49:42.829378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:42.828785    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62" podUID="1137058f-bb14-42e6-ac55-d2104536ffcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 09:49:43.829930 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:43.829895    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 09:49:43.830528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:43.830496    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62" podUID="1137058f-bb14-42e6-ac55-d2104536ffcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 09:49:44.831453 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:44.831418    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:44.831842 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:44.831719    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:44.832605 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:44.832586    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:49:44.860718 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:44.860666    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podStartSLOduration=5.860648467 podStartE2EDuration="5.860648467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:49:41.042930189 +0000 UTC m=+861.875588270" watchObservedRunningTime="2026-11-02 09:49:44.860648467 +0000 UTC m=+865.693306549"
Nov 02 09:49:48.161376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:48.160688    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:49.098376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:49.093630    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:49.160950 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:49.160897    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 02 09:49:49.162580 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:49.162556    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:49:49.452465 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:49.423439    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 09:49:49.629504 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:49.629457    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:49.796133 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:49.796104    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:49.928921 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:49.928890    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:50.199205 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:50.198707    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:49:50.225331 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:50.223923    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:49:50.513446 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:50.513418    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:49:51.542589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.542549    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"]
Nov 02 09:49:51.586698 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.586663    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"]
Nov 02 09:49:51.587027 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.587008    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:51.604334 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.604289    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-plugins-e2e-extmod-b-204fba\""
Nov 02 09:49:51.604656 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.604638    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"routing-overlay\""
Nov 02 09:49:51.604907 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.604890    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"openshift-service-ca.crt\""
Nov 02 09:49:51.605072 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.605059    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"kube-root-ca.crt\""
Nov 02 09:49:51.639549 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.639519    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-11e3267f4507\""
Nov 02 09:49:51.642742 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.642550    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-e2e--e7669bbb-dockercfg-zngpv\""
Nov 02 09:49:51.642891 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.642640    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-4a99520c97db\""
Nov 02 09:49:51.642992 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.642669    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-1521404c1537\""
Nov 02 09:49:51.643075 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.642694    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-e0607e4dc0eb\""
Nov 02 09:49:51.738000 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.737959    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h78m\" (UniqueName: \"kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-kube-api-access-4h78m\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:51.738199 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.738137    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:51.738262 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.738197    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e481843-a778-4bd4-aee5-42bf2b15f805-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:51.738321 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.738294    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e481843-a778-4bd4-aee5-42bf2b15f805-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:51.839206 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.839114    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:51.839206 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.839186    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e481843-a778-4bd4-aee5-42bf2b15f805-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:51.839501 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.839238    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e481843-a778-4bd4-aee5-42bf2b15f805-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:51.839501 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.839312    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h78m\" (UniqueName: \"kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-kube-api-access-4h78m\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:51.840255 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.840229    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e481843-a778-4bd4-aee5-42bf2b15f805-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:51.840403 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.840280    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e481843-a778-4bd4-aee5-42bf2b15f805-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:51.842841 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.842818    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:51.851943 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.851922    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h78m\" (UniqueName: \"kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-kube-api-access-4h78m\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:51.928496 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.928464    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"]
Nov 02 09:49:51.933739 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.933713    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:51.939014 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.938987    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"openshift-service-ca.crt\""
Nov 02 09:49:51.939152 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.939110    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-plugins-e2e-extmod-a-629129\""
Nov 02 09:49:51.939152 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.939149    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-cedce8216574\""
Nov 02 09:49:51.939268 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.939235    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"kube-root-ca.crt\""
Nov 02 09:49:51.939318 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.939295    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"routing-overlay\""
Nov 02 09:49:51.942806 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.942781    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"]
Nov 02 09:49:51.956179 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:51.956149    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:52.042751 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.042182    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:52.042751 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.042335    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2dmw\" (UniqueName: \"kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-kube-api-access-j2dmw\") pod \"payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:52.042751 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.042403    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:52.042751 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.042462    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:52.144254 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.144164    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:52.144474 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.144307    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2dmw\" (UniqueName: \"kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-kube-api-access-j2dmw\") pod \"payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:52.144474 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.144345    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:52.144474 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.144431    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:52.145230 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.145205    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:52.145324 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.145251    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:52.146986 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.146962    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:52.156823 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.156802    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2dmw\" (UniqueName: \"kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-kube-api-access-j2dmw\") pod \"payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:52.254280 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.253953    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:52.404774 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.404741    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"]
Nov 02 09:49:52.487689 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.487619    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p" event={"ID":"4e481843-a778-4bd4-aee5-42bf2b15f805","Type":"ContainerStarted","Data":"7d050c63149ccd41d1f11115c417b6bdcb65bea7c1b84ccb521cb729a6de8429"}
Nov 02 09:49:52.728706 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:49:52.728669    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fe9de5_b2b8_4af5_8a91_08b0d0826e6e.slice/crio-c9c5612a8e062cd93aec9dc091d3fd828b40a36b0648862eccb27d3ba1080f92 WatchSource:0}: Error finding container c9c5612a8e062cd93aec9dc091d3fd828b40a36b0648862eccb27d3ba1080f92: Status 404 returned error can't find the container with id c9c5612a8e062cd93aec9dc091d3fd828b40a36b0648862eccb27d3ba1080f92
Nov 02 09:49:52.729511 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:52.729479    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"]
Nov 02 09:49:53.278409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:53.269303    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"]
Nov 02 09:49:53.539454 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:53.531286    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x" event={"ID":"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e","Type":"ContainerStarted","Data":"e26c093f684b164c0fdfcf1ec25f330572bc8c2e735f06806ee77846d2268b3f"}
Nov 02 09:49:53.539454 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:53.531422    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x" event={"ID":"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e","Type":"ContainerStarted","Data":"c9c5612a8e062cd93aec9dc091d3fd828b40a36b0648862eccb27d3ba1080f92"}
Nov 02 09:49:53.539454 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:53.532299    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:49:53.556376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:53.553470    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p" event={"ID":"4e481843-a778-4bd4-aee5-42bf2b15f805","Type":"ContainerStarted","Data":"5019f50d0aa42a5eab80c1b90e90bca5598feee0a9feb46d0501d9a3ba593b82"}
Nov 02 09:49:53.556376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:53.554235    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:49:53.602534 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:53.594657    4614 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-plugins-e2e-extmod-a-629129: configmap "payload-processing-external-model-plugins-e2e-extmod-a-629129" not found
Nov 02 09:49:53.602534 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:53.594737    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume podName:b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e nodeName:}" failed. No retries permitted until 2026-11-02 09:49:54.094717099 +0000 UTC m=+874.927375171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x" (UID: "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-629129" not found
Nov 02 09:49:53.602534 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:53.595290    4614 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-cedce8216574: secret "agc-cred-cedce8216574" not found
Nov 02 09:49:53.602534 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:53.595320    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x: secret "agc-cred-cedce8216574" not found
Nov 02 09:49:53.602534 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:53.595399    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials podName:b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e nodeName:}" failed. No retries permitted until 2026-11-02 09:49:54.095374439 +0000 UTC m=+874.928032510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x" (UID: "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e") : secret "agc-cred-cedce8216574" not found
Nov 02 09:49:53.634798 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:53.634727    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x" podStartSLOduration=2.634705964 podStartE2EDuration="2.634705964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:49:53.561811361 +0000 UTC m=+874.394469458" watchObservedRunningTime="2026-11-02 09:49:53.634705964 +0000 UTC m=+874.467364045"
Nov 02 09:49:53.651186 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:53.651122    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p" podStartSLOduration=2.65110234 podStartE2EDuration="2.65110234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:49:53.608710026 +0000 UTC m=+874.441368122" watchObservedRunningTime="2026-11-02 09:49:53.65110234 +0000 UTC m=+874.483760422"
Nov 02 09:49:53.697507 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:53.696178    4614 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-e0607e4dc0eb: secret "agc-cred-e0607e4dc0eb" not found
Nov 02 09:49:53.697507 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:53.696207    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p: secret "agc-cred-e0607e4dc0eb" not found
Nov 02 09:49:53.697507 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:53.696278    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials podName:4e481843-a778-4bd4-aee5-42bf2b15f805 nodeName:}" failed. No retries permitted until 2026-11-02 09:49:54.196258018 +0000 UTC m=+875.028916080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p" (UID: "4e481843-a778-4bd4-aee5-42bf2b15f805") : secret "agc-cred-e0607e4dc0eb" not found
Nov 02 09:49:53.974856 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:53.974823    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"]
Nov 02 09:49:54.104293 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:54.104252    4614 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-plugins-e2e-extmod-a-629129: configmap "payload-processing-external-model-plugins-e2e-extmod-a-629129" not found
Nov 02 09:49:54.104293 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:54.104287    4614 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-cedce8216574: secret "agc-cred-cedce8216574" not found
Nov 02 09:49:54.104539 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:54.104307    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x: secret "agc-cred-cedce8216574" not found
Nov 02 09:49:54.104539 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:54.104347    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume podName:b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e nodeName:}" failed. No retries permitted until 2026-11-02 09:49:55.104331105 +0000 UTC m=+875.936989167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x" (UID: "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-629129" not found
Nov 02 09:49:54.104539 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:54.104395    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials podName:b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e nodeName:}" failed. No retries permitted until 2026-11-02 09:49:55.104387294 +0000 UTC m=+875.937045352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x" (UID: "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e") : secret "agc-cred-cedce8216574" not found
Nov 02 09:49:54.205655 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:54.205623    4614 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-e0607e4dc0eb: secret "agc-cred-e0607e4dc0eb" not found
Nov 02 09:49:54.205655 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:54.205649    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p: secret "agc-cred-e0607e4dc0eb" not found
Nov 02 09:49:54.205847 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:54.205705    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials podName:4e481843-a778-4bd4-aee5-42bf2b15f805 nodeName:}" failed. No retries permitted until 2026-11-02 09:49:55.2056908 +0000 UTC m=+876.038348874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p" (UID: "4e481843-a778-4bd4-aee5-42bf2b15f805") : secret "agc-cred-e0607e4dc0eb" not found
Nov 02 09:49:55.124817 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:55.124780    4614 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-plugins-e2e-extmod-a-629129: configmap "payload-processing-external-model-plugins-e2e-extmod-a-629129" not found
Nov 02 09:49:55.125263 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:55.124863    4614 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-cedce8216574: secret "agc-cred-cedce8216574" not found
Nov 02 09:49:55.125263 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:55.124873    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume podName:b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e nodeName:}" failed. No retries permitted until 2026-11-02 09:49:57.124852017 +0000 UTC m=+877.957510094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x" (UID: "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-629129" not found
Nov 02 09:49:55.125263 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:55.124881    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x: secret "agc-cred-cedce8216574" not found
Nov 02 09:49:55.125263 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:55.124918    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials podName:b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e nodeName:}" failed. No retries permitted until 2026-11-02 09:49:57.12490741 +0000 UTC m=+877.957565473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x" (UID: "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e") : secret "agc-cred-cedce8216574" not found
Nov 02 09:49:55.226450 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:55.226421    4614 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-e0607e4dc0eb: secret "agc-cred-e0607e4dc0eb" not found
Nov 02 09:49:55.226450 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:55.226445    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p: secret "agc-cred-e0607e4dc0eb" not found
Nov 02 09:49:55.226660 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:55.226493    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials podName:4e481843-a778-4bd4-aee5-42bf2b15f805 nodeName:}" failed. No retries permitted until 2026-11-02 09:49:57.226480949 +0000 UTC m=+878.059139007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p" (UID: "4e481843-a778-4bd4-aee5-42bf2b15f805") : secret "agc-cred-e0607e4dc0eb" not found
Nov 02 09:49:57.167642 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:57.167603    4614 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-629129/agc-cred-cedce8216574: secret "agc-cred-cedce8216574" not found
Nov 02 09:49:57.167642 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:57.167633    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x: secret "agc-cred-cedce8216574" not found
Nov 02 09:49:57.168182 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:57.167680    4614 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-plugins-e2e-extmod-a-629129: configmap "payload-processing-external-model-plugins-e2e-extmod-a-629129" not found
Nov 02 09:49:57.168182 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:57.167702    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials podName:b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e nodeName:}" failed. No retries permitted until 2026-11-02 09:50:01.167683504 +0000 UTC m=+882.000341588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x" (UID: "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e") : secret "agc-cred-cedce8216574" not found
Nov 02 09:49:57.168182 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:57.167724    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume podName:b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e nodeName:}" failed. No retries permitted until 2026-11-02 09:50:01.167713006 +0000 UTC m=+882.000371079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x" (UID: "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-629129" not found
Nov 02 09:49:57.269301 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:57.269267    4614 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-e0607e4dc0eb: secret "agc-cred-e0607e4dc0eb" not found
Nov 02 09:49:57.269301 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:57.269295    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p: secret "agc-cred-e0607e4dc0eb" not found
Nov 02 09:49:57.269564 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:49:57.269388    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials podName:4e481843-a778-4bd4-aee5-42bf2b15f805 nodeName:}" failed. No retries permitted until 2026-11-02 09:50:01.269342855 +0000 UTC m=+882.102000912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p" (UID: "4e481843-a778-4bd4-aee5-42bf2b15f805") : secret "agc-cred-e0607e4dc0eb" not found
Nov 02 09:49:59.590420 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:59.587431    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x" podUID="b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e" containerName="payload-processing" containerID="cri-o://e26c093f684b164c0fdfcf1ec25f330572bc8c2e735f06806ee77846d2268b3f" gracePeriod=25
Nov 02 09:49:59.915862 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:49:59.915834    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:50:00.012607 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.012569    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials\") pod \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") "
Nov 02 09:50:00.012796 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.012658    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-routing-overlay-volume\") pod \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") "
Nov 02 09:50:00.012796 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.012729    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-kube-api-access-j2dmw\" (UniqueName: \"kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-kube-api-access-j2dmw\") pod \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") "
Nov 02 09:50:00.012922 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.012897    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume\") pod \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\" (UID: \"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e\") "
Nov 02 09:50:00.013109 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.013029    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-routing-overlay-volume" pod "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e" (UID: "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:00.013222 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.013201    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume" pod "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e" (UID: "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:00.013413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.013396    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:00.013485 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.013413    4614 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-routing-overlay-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:00.014774 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.014743    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials" pod "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e" (UID: "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:00.015254 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.015229    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-kube-api-access-j2dmw" pod "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e" (UID: "b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e"). InnerVolumeSpecName "kube-api-access-j2dmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:00.119375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.117646    4614 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-provider-credentials\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:00.119375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.117687    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2dmw\" (UniqueName: \"kubernetes.io/projected/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e-kube-api-access-j2dmw\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:00.606128 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.606087    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p" podUID="4e481843-a778-4bd4-aee5-42bf2b15f805" containerName="payload-processing" containerID="cri-o://5019f50d0aa42a5eab80c1b90e90bca5598feee0a9feb46d0501d9a3ba593b82" gracePeriod=25
Nov 02 09:50:00.672061 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.672027    4614 generic.go:356] "Generic (PLEG): container finished" podID="b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e" containerID="e26c093f684b164c0fdfcf1ec25f330572bc8c2e735f06806ee77846d2268b3f" exitCode=0
Nov 02 09:50:00.672223 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.672096    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"
Nov 02 09:50:00.672223 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.672119    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x" event={"ID":"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e","Type":"ContainerDied","Data":"e26c093f684b164c0fdfcf1ec25f330572bc8c2e735f06806ee77846d2268b3f"}
Nov 02 09:50:00.672223 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.672176    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x" event={"ID":"b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e","Type":"ContainerDied","Data":"c9c5612a8e062cd93aec9dc091d3fd828b40a36b0648862eccb27d3ba1080f92"}
Nov 02 09:50:00.672223 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.672200    4614 scope.go:122] "RemoveContainer" containerID="e26c093f684b164c0fdfcf1ec25f330572bc8c2e735f06806ee77846d2268b3f"
Nov 02 09:50:00.721044 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.721017    4614 scope.go:122] "RemoveContainer" containerID="e26c093f684b164c0fdfcf1ec25f330572bc8c2e735f06806ee77846d2268b3f"
Nov 02 09:50:00.721412 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:50:00.721390    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26c093f684b164c0fdfcf1ec25f330572bc8c2e735f06806ee77846d2268b3f\": container with ID starting with e26c093f684b164c0fdfcf1ec25f330572bc8c2e735f06806ee77846d2268b3f not found: ID does not exist" containerID="e26c093f684b164c0fdfcf1ec25f330572bc8c2e735f06806ee77846d2268b3f"
Nov 02 09:50:00.721488 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.721422    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26c093f684b164c0fdfcf1ec25f330572bc8c2e735f06806ee77846d2268b3f"} err="failed to get container status \"e26c093f684b164c0fdfcf1ec25f330572bc8c2e735f06806ee77846d2268b3f\": rpc error: code = NotFound desc = could not find container \"e26c093f684b164c0fdfcf1ec25f330572bc8c2e735f06806ee77846d2268b3f\": container with ID starting with e26c093f684b164c0fdfcf1ec25f330572bc8c2e735f06806ee77846d2268b3f not found: ID does not exist"
Nov 02 09:50:00.730912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.730883    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"]
Nov 02 09:50:00.744987 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:00.741920    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-d5c77l68x"]
Nov 02 09:50:01.066142 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.066120    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:50:01.143750 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.143714    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e481843-a778-4bd4-aee5-42bf2b15f805-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e481843-a778-4bd4-aee5-42bf2b15f805-plugins-config-volume\") pod \"4e481843-a778-4bd4-aee5-42bf2b15f805\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") "
Nov 02 09:50:01.143945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.143777    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e481843-a778-4bd4-aee5-42bf2b15f805-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e481843-a778-4bd4-aee5-42bf2b15f805-routing-overlay-volume\") pod \"4e481843-a778-4bd4-aee5-42bf2b15f805\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") "
Nov 02 09:50:01.143945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.143881    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials\") pod \"4e481843-a778-4bd4-aee5-42bf2b15f805\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") "
Nov 02 09:50:01.143945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.143913    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-kube-api-access-4h78m\" (UniqueName: \"kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-kube-api-access-4h78m\") pod \"4e481843-a778-4bd4-aee5-42bf2b15f805\" (UID: \"4e481843-a778-4bd4-aee5-42bf2b15f805\") "
Nov 02 09:50:01.144460 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.144258    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e481843-a778-4bd4-aee5-42bf2b15f805-plugins-config-volume" pod "4e481843-a778-4bd4-aee5-42bf2b15f805" (UID: "4e481843-a778-4bd4-aee5-42bf2b15f805"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:01.144460 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.144309    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e481843-a778-4bd4-aee5-42bf2b15f805-routing-overlay-volume" pod "4e481843-a778-4bd4-aee5-42bf2b15f805" (UID: "4e481843-a778-4bd4-aee5-42bf2b15f805"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:01.144608 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.144494    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e481843-a778-4bd4-aee5-42bf2b15f805-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:01.144608 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.144509    4614 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e481843-a778-4bd4-aee5-42bf2b15f805-routing-overlay-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:01.146308 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.146277    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials" pod "4e481843-a778-4bd4-aee5-42bf2b15f805" (UID: "4e481843-a778-4bd4-aee5-42bf2b15f805"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:01.146605 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.146587    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-kube-api-access-4h78m" pod "4e481843-a778-4bd4-aee5-42bf2b15f805" (UID: "4e481843-a778-4bd4-aee5-42bf2b15f805"). InnerVolumeSpecName "kube-api-access-4h78m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:01.246012 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.245972    4614 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-provider-credentials\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:01.246012 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.246000    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h78m\" (UniqueName: \"kubernetes.io/projected/4e481843-a778-4bd4-aee5-42bf2b15f805-kube-api-access-4h78m\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:01.339508 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.339407    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e" path="/var/lib/kubelet/pods/b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e/volumes"
Nov 02 09:50:01.741423 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.738787    4614 generic.go:356] "Generic (PLEG): container finished" podID="4e481843-a778-4bd4-aee5-42bf2b15f805" containerID="5019f50d0aa42a5eab80c1b90e90bca5598feee0a9feb46d0501d9a3ba593b82" exitCode=0
Nov 02 09:50:01.741423 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.738835    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p" event={"ID":"4e481843-a778-4bd4-aee5-42bf2b15f805","Type":"ContainerDied","Data":"5019f50d0aa42a5eab80c1b90e90bca5598feee0a9feb46d0501d9a3ba593b82"}
Nov 02 09:50:01.741423 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.738882    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p" event={"ID":"4e481843-a778-4bd4-aee5-42bf2b15f805","Type":"ContainerDied","Data":"7d050c63149ccd41d1f11115c417b6bdcb65bea7c1b84ccb521cb729a6de8429"}
Nov 02 09:50:01.741423 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.738909    4614 scope.go:122] "RemoveContainer" containerID="5019f50d0aa42a5eab80c1b90e90bca5598feee0a9feb46d0501d9a3ba593b82"
Nov 02 09:50:01.741423 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.739052    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"
Nov 02 09:50:01.770603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.767058    4614 scope.go:122] "RemoveContainer" containerID="5019f50d0aa42a5eab80c1b90e90bca5598feee0a9feb46d0501d9a3ba593b82"
Nov 02 09:50:01.770603 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:50:01.767460    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5019f50d0aa42a5eab80c1b90e90bca5598feee0a9feb46d0501d9a3ba593b82\": container with ID starting with 5019f50d0aa42a5eab80c1b90e90bca5598feee0a9feb46d0501d9a3ba593b82 not found: ID does not exist" containerID="5019f50d0aa42a5eab80c1b90e90bca5598feee0a9feb46d0501d9a3ba593b82"
Nov 02 09:50:01.770603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.767496    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5019f50d0aa42a5eab80c1b90e90bca5598feee0a9feb46d0501d9a3ba593b82"} err="failed to get container status \"5019f50d0aa42a5eab80c1b90e90bca5598feee0a9feb46d0501d9a3ba593b82\": rpc error: code = NotFound desc = could not find container \"5019f50d0aa42a5eab80c1b90e90bca5598feee0a9feb46d0501d9a3ba593b82\": container with ID starting with 5019f50d0aa42a5eab80c1b90e90bca5598feee0a9feb46d0501d9a3ba593b82 not found: ID does not exist"
Nov 02 09:50:01.797377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.792421    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"]
Nov 02 09:50:01.949377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:01.945894    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-85887rd8p"]
Nov 02 09:50:03.350336 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:03.350297    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e481843-a778-4bd4-aee5-42bf2b15f805" path="/var/lib/kubelet/pods/4e481843-a778-4bd4-aee5-42bf2b15f805/volumes"
Nov 02 09:50:05.499571 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.499534    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"]
Nov 02 09:50:05.500057 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.500042    4614 cpu_manager.go:414] "containerMap: removing container" podUID="4e481843-a778-4bd4-aee5-42bf2b15f805" containerName="payload-processing"
Nov 02 09:50:05.500120 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.500057    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e481843-a778-4bd4-aee5-42bf2b15f805" containerName="payload-processing"
Nov 02 09:50:05.500120 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.500070    4614 cpu_manager.go:414] "containerMap: removing container" podUID="b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e" containerName="payload-processing"
Nov 02 09:50:05.500120 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.500076    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e" containerName="payload-processing"
Nov 02 09:50:05.500209 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.500162    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e481843-a778-4bd4-aee5-42bf2b15f805" containerName="payload-processing"
Nov 02 09:50:05.500209 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.500171    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9fe9de5-b2b8-4af5-8a91-08b0d0826e6e" containerName="payload-processing"
Nov 02 09:50:05.503529 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.503509    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:05.512938 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.512914    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"]
Nov 02 09:50:05.518734 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.518644    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-plugins-e2e-extmod-b-204fba\""
Nov 02 09:50:05.518734 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.518644    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-8960fbffc2fc\""
Nov 02 09:50:05.518734 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.518679    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"kube-root-ca.crt\""
Nov 02 09:50:05.518734 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.518680    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-4a99520c97db\""
Nov 02 09:50:05.518734 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.518645    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-e2e--e7669bbb-dockercfg-zngpv\""
Nov 02 09:50:05.518734 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.518686    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-11e3267f4507\""
Nov 02 09:50:05.519176 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.519096    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"routing-overlay\""
Nov 02 09:50:05.519409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.519389    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"openshift-service-ca.crt\""
Nov 02 09:50:05.519602 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.519580    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-1521404c1537\""
Nov 02 09:50:05.522944 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.522908    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cfc66550-db37-4578-a682-3155465834f2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:05.523068 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.522992    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cfc66550-db37-4578-a682-3155465834f2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:05.523068 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.523034    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc66550-db37-4578-a682-3155465834f2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:05.523190 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.523156    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qd7v\" (UniqueName: \"kubernetes.io/projected/cfc66550-db37-4578-a682-3155465834f2-kube-api-access-9qd7v\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:05.623938 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.623904    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qd7v\" (UniqueName: \"kubernetes.io/projected/cfc66550-db37-4578-a682-3155465834f2-kube-api-access-9qd7v\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:05.624124 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.623992    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cfc66550-db37-4578-a682-3155465834f2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:05.624124 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.624036    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cfc66550-db37-4578-a682-3155465834f2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:05.624124 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.624066    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc66550-db37-4578-a682-3155465834f2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:05.624743 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.624709    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cfc66550-db37-4578-a682-3155465834f2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:05.624863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.624795    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc66550-db37-4578-a682-3155465834f2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:05.626807 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.626786    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cfc66550-db37-4578-a682-3155465834f2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:05.636854 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.636833    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qd7v\" (UniqueName: \"kubernetes.io/projected/cfc66550-db37-4578-a682-3155465834f2-kube-api-access-9qd7v\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:05.818300 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:05.818221    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:06.106337 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:06.106293    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"]
Nov 02 09:50:06.106532 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:50:06.106440    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc66550_db37_4578_a682_3155465834f2.slice/crio-02854b88533dc4b0273fc57c9255caaa432321e0719381c26012c70580ae5da7 WatchSource:0}: Error finding container 02854b88533dc4b0273fc57c9255caaa432321e0719381c26012c70580ae5da7: Status 404 returned error can't find the container with id 02854b88533dc4b0273fc57c9255caaa432321e0719381c26012c70580ae5da7
Nov 02 09:50:06.838527 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:06.838487    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp" event={"ID":"cfc66550-db37-4578-a682-3155465834f2","Type":"ContainerStarted","Data":"f96b56ee8cdc8c45ffd5d7bf45a96bd34ee987cd964189a0e21351386b40fd7d"}
Nov 02 09:50:06.838942 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:06.838545    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp" event={"ID":"cfc66550-db37-4578-a682-3155465834f2","Type":"ContainerStarted","Data":"02854b88533dc4b0273fc57c9255caaa432321e0719381c26012c70580ae5da7"}
Nov 02 09:50:06.838942 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:06.838583    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:06.860242 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:06.860187    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp" podStartSLOduration=1.860175178 podStartE2EDuration="1.860175178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:50:06.858333459 +0000 UTC m=+887.690991540" watchObservedRunningTime="2026-11-02 09:50:06.860175178 +0000 UTC m=+887.692833260"
Nov 02 09:50:09.636769 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:50:09.636734    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:50:15.730761 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:15.730713    4614 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:17.060910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:17.059451    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:17.091376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:17.090462    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:17.239611 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:17.239562    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:17.343562 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:17.343466    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:50:17.344787 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:17.344749    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:17.427563 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:17.427504    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 02 09:50:17.428309 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:17.427824    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:17.871379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:17.833494    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"]
Nov 02 09:50:17.899104 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:17.891040    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd" podUID="a5ed6879-0ce6-4705-a1bc-eefbfef9157f" containerName="istio-proxy" containerID="cri-o://9a4878af8a305f63a68fbede3104f2dd2da02360caf6b238e2c4343d2c694363" gracePeriod=30
Nov 02 09:50:17.899104 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:17.890769    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:50:17.899104 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:17.891225    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:18.423829 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:18.286209    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:18.423829 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:18.287849    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:50:18.423829 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:18.360880    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" podUID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:18.423829 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:18.361728    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:50:18.423829 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:18.361992    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:18.510408 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:18.507753    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 02 09:50:18.510408 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:18.508319    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 09:50:18.510408 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:18.508903    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:50:18.510408 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:18.509101    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:50:18.543376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:18.535301    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:18.543376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:18.537178    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:50:19.025143 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:19.024411    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 02 09:50:19.103412 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:19.071260    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:50:19.135378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:19.104275    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 02 09:50:19.175087 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:19.175041    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:50:19.295188 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:19.295088    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:19.300097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:19.300071    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:50:19.338614 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:19.336632    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:50:19.413380 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:19.392830    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:50:19.502651 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:19.502623    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:50:19.626997 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:19.626969    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 09:50:19.629364 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:19.629333    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 09:50:23.085515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.085488    4614 generic.go:356] "Generic (PLEG): container finished" podID="a5ed6879-0ce6-4705-a1bc-eefbfef9157f" containerID="9a4878af8a305f63a68fbede3104f2dd2da02360caf6b238e2c4343d2c694363" exitCode=0
Nov 02 09:50:23.085862 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.085526    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd" event={"ID":"a5ed6879-0ce6-4705-a1bc-eefbfef9157f","Type":"ContainerDied","Data":"9a4878af8a305f63a68fbede3104f2dd2da02360caf6b238e2c4343d2c694363"}
Nov 02 09:50:23.215583 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.215559    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:50:23.326870 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.326781    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istiod-ca-cert\") pod \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") "
Nov 02 09:50:23.326870 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.326830    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-token\" (UniqueName: \"kubernetes.io/projected/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-token\") pod \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") "
Nov 02 09:50:23.327071 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.326888    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-podinfo\") pod \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") "
Nov 02 09:50:23.327071 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.326923    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-data\") pod \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") "
Nov 02 09:50:23.327071 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.326960    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-workload-socket\") pod \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") "
Nov 02 09:50:23.327071 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.327016    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-workload-certs\") pod \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") "
Nov 02 09:50:23.327199 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.327108    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-envoy\") pod \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") "
Nov 02 09:50:23.327199 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.327180    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-kube-api-access-dvbbr\" (UniqueName: \"kubernetes.io/projected/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-kube-api-access-dvbbr\") pod \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") "
Nov 02 09:50:23.327273 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.327258    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-credential-socket\") pod \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\" (UID: \"a5ed6879-0ce6-4705-a1bc-eefbfef9157f\") "
Nov 02 09:50:23.327309 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.327275    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-data" pod "a5ed6879-0ce6-4705-a1bc-eefbfef9157f" (UID: "a5ed6879-0ce6-4705-a1bc-eefbfef9157f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:23.327811 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.327786    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.327960 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.327820    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-workload-certs" pod "a5ed6879-0ce6-4705-a1bc-eefbfef9157f" (UID: "a5ed6879-0ce6-4705-a1bc-eefbfef9157f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:23.327960 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.327876    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istiod-ca-cert" pod "a5ed6879-0ce6-4705-a1bc-eefbfef9157f" (UID: "a5ed6879-0ce6-4705-a1bc-eefbfef9157f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:23.328092 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.328010    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-credential-socket" pod "a5ed6879-0ce6-4705-a1bc-eefbfef9157f" (UID: "a5ed6879-0ce6-4705-a1bc-eefbfef9157f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:23.328092 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.328032    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-workload-socket" pod "a5ed6879-0ce6-4705-a1bc-eefbfef9157f" (UID: "a5ed6879-0ce6-4705-a1bc-eefbfef9157f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:23.330405 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.330338    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-podinfo" pod "a5ed6879-0ce6-4705-a1bc-eefbfef9157f" (UID: "a5ed6879-0ce6-4705-a1bc-eefbfef9157f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:50:23.330549 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.330343    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-token" pod "a5ed6879-0ce6-4705-a1bc-eefbfef9157f" (UID: "a5ed6879-0ce6-4705-a1bc-eefbfef9157f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:23.330634 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.330611    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-envoy" pod "a5ed6879-0ce6-4705-a1bc-eefbfef9157f" (UID: "a5ed6879-0ce6-4705-a1bc-eefbfef9157f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:23.330826 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.330798    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-kube-api-access-dvbbr" pod "a5ed6879-0ce6-4705-a1bc-eefbfef9157f" (UID: "a5ed6879-0ce6-4705-a1bc-eefbfef9157f"). InnerVolumeSpecName "kube-api-access-dvbbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:23.428998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.428967    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.428998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.428996    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvbbr\" (UniqueName: \"kubernetes.io/projected/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-kube-api-access-dvbbr\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.429200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.429008    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.429200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.429018    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.429200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.429030    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.429200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.429042    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.429200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.429051    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:23.429200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:23.429059    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ed6879-0ce6-4705-a1bc-eefbfef9157f-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:24.093500 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:24.093464    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd" event={"ID":"a5ed6879-0ce6-4705-a1bc-eefbfef9157f","Type":"ContainerDied","Data":"06e6b969382a62622b03ad7e1e7b38b64bde75a0d703b507d9c50daaa0b823eb"}
Nov 02 09:50:24.093500 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:24.093496    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"
Nov 02 09:50:24.093989 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:24.093539    4614 scope.go:122] "RemoveContainer" containerID="9a4878af8a305f63a68fbede3104f2dd2da02360caf6b238e2c4343d2c694363"
Nov 02 09:50:24.118886 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:24.118857    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"]
Nov 02 09:50:24.125401 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:24.125377    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-301be8-openshift-default-bf4f59db5-66ssd"]
Nov 02 09:50:24.869297 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:24.869265    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"]
Nov 02 09:50:25.348464 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:25.346790    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ed6879-0ce6-4705-a1bc-eefbfef9157f" path="/var/lib/kubelet/pods/a5ed6879-0ce6-4705-a1bc-eefbfef9157f/volumes"
Nov 02 09:50:28.224528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:28.224477    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:28.299501 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:28.299446    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 02 09:50:28.332388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:28.331442    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:28.404505 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:28.404452    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:28.625879 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:28.625784    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:28.775494 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:28.775439    4614 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:29.227508 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:29.227458    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 02 09:50:29.228879 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:29.228852    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:50:29.451490 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:29.451395    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:29.461387 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:29.451981    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:50:29.461387 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:29.457559    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 02 09:50:29.468382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:29.464646    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:50:29.509120 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:29.509022    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:50:29.658527 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:29.626469    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 02 09:50:29.844464 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:29.843100    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:50:29.844464 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:29.843163    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:50:29.844464 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:29.843194    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:50:29.870287 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:29.870111    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr" podUID="29b9d779-f66b-4283-8eef-d1f80751fef1" containerName="payload-pre-processing" containerID="cri-o://cc912e937b666b975f1a8c1db05a019241f44868b89f376d9d029b32e0c4f26e" gracePeriod=25
Nov 02 09:50:29.908941 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:29.900179    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:50:30.108184 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:30.108101    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:50:30.253521 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:30.253479    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr" podUID="29b9d779-f66b-4283-8eef-d1f80751fef1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:50:39.033420 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:39.033386    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"]
Nov 02 09:50:39.037430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:39.037398    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b708f2bf-jtfbj"]
Nov 02 09:50:39.182503 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:39.182470    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"]
Nov 02 09:50:39.182790 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:39.182759    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm" podUID="270f7540-a28f-446a-bc7f-3b3d143bc351" containerName="istio-proxy" containerID="cri-o://2fc1c464b1f5f9d7b6852b2d94d1a420d24d1b2c1aa66ea970cf93dcfc3f96ce" gracePeriod=30
Nov 02 09:50:39.323248 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:39.323155    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2da5ca-78ee-4a03-8e4b-7ee199593b6f" path="/var/lib/kubelet/pods/4a2da5ca-78ee-4a03-8e4b-7ee199593b6f/volumes"
Nov 02 09:50:39.643073 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:50:39.642973    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:50:40.228242 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:40.228191    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr" podUID="29b9d779-f66b-4283-8eef-d1f80751fef1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 09:50:44.506931 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.506904    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:50:44.551810 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.551777    4614 generic.go:356] "Generic (PLEG): container finished" podID="270f7540-a28f-446a-bc7f-3b3d143bc351" containerID="2fc1c464b1f5f9d7b6852b2d94d1a420d24d1b2c1aa66ea970cf93dcfc3f96ce" exitCode=0
Nov 02 09:50:44.551984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.551826    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"
Nov 02 09:50:44.551984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.551842    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm" event={"ID":"270f7540-a28f-446a-bc7f-3b3d143bc351","Type":"ContainerDied","Data":"2fc1c464b1f5f9d7b6852b2d94d1a420d24d1b2c1aa66ea970cf93dcfc3f96ce"}
Nov 02 09:50:44.551984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.551892    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm" event={"ID":"270f7540-a28f-446a-bc7f-3b3d143bc351","Type":"ContainerDied","Data":"9128ccd490161322e3b33ba7cb45ba4b85d2a10ba4f3f032446394c2adaf020c"}
Nov 02 09:50:44.551984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.551923    4614 scope.go:122] "RemoveContainer" containerID="2fc1c464b1f5f9d7b6852b2d94d1a420d24d1b2c1aa66ea970cf93dcfc3f96ce"
Nov 02 09:50:44.562147 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.562126    4614 scope.go:122] "RemoveContainer" containerID="2fc1c464b1f5f9d7b6852b2d94d1a420d24d1b2c1aa66ea970cf93dcfc3f96ce"
Nov 02 09:50:44.562458 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:50:44.562436    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc1c464b1f5f9d7b6852b2d94d1a420d24d1b2c1aa66ea970cf93dcfc3f96ce\": container with ID starting with 2fc1c464b1f5f9d7b6852b2d94d1a420d24d1b2c1aa66ea970cf93dcfc3f96ce not found: ID does not exist" containerID="2fc1c464b1f5f9d7b6852b2d94d1a420d24d1b2c1aa66ea970cf93dcfc3f96ce"
Nov 02 09:50:44.562585 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.562464    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc1c464b1f5f9d7b6852b2d94d1a420d24d1b2c1aa66ea970cf93dcfc3f96ce"} err="failed to get container status \"2fc1c464b1f5f9d7b6852b2d94d1a420d24d1b2c1aa66ea970cf93dcfc3f96ce\": rpc error: code = NotFound desc = could not find container \"2fc1c464b1f5f9d7b6852b2d94d1a420d24d1b2c1aa66ea970cf93dcfc3f96ce\": container with ID starting with 2fc1c464b1f5f9d7b6852b2d94d1a420d24d1b2c1aa66ea970cf93dcfc3f96ce not found: ID does not exist"
Nov 02 09:50:44.604040 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604012    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-workload-certs\") pod \"270f7540-a28f-446a-bc7f-3b3d143bc351\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") "
Nov 02 09:50:44.604166 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604073    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-credential-socket\") pod \"270f7540-a28f-446a-bc7f-3b3d143bc351\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") "
Nov 02 09:50:44.604166 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604140    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-workload-socket\") pod \"270f7540-a28f-446a-bc7f-3b3d143bc351\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") "
Nov 02 09:50:44.604235 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604173    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-token\" (UniqueName: \"kubernetes.io/projected/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-token\") pod \"270f7540-a28f-446a-bc7f-3b3d143bc351\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") "
Nov 02 09:50:44.604235 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604198    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-podinfo\") pod \"270f7540-a28f-446a-bc7f-3b3d143bc351\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") "
Nov 02 09:50:44.604342 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604257    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-data\") pod \"270f7540-a28f-446a-bc7f-3b3d143bc351\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") "
Nov 02 09:50:44.604342 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604314    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-envoy\") pod \"270f7540-a28f-446a-bc7f-3b3d143bc351\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") "
Nov 02 09:50:44.604468 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604365    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270f7540-a28f-446a-bc7f-3b3d143bc351-kube-api-access-k8sqd\" (UniqueName: \"kubernetes.io/projected/270f7540-a28f-446a-bc7f-3b3d143bc351-kube-api-access-k8sqd\") pod \"270f7540-a28f-446a-bc7f-3b3d143bc351\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") "
Nov 02 09:50:44.604468 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604435    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/270f7540-a28f-446a-bc7f-3b3d143bc351-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270f7540-a28f-446a-bc7f-3b3d143bc351-istiod-ca-cert\") pod \"270f7540-a28f-446a-bc7f-3b3d143bc351\" (UID: \"270f7540-a28f-446a-bc7f-3b3d143bc351\") "
Nov 02 09:50:44.604596 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604346    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-workload-certs" pod "270f7540-a28f-446a-bc7f-3b3d143bc351" (UID: "270f7540-a28f-446a-bc7f-3b3d143bc351"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:44.604596 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604428    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-credential-socket" pod "270f7540-a28f-446a-bc7f-3b3d143bc351" (UID: "270f7540-a28f-446a-bc7f-3b3d143bc351"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:44.604596 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604570    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-data" pod "270f7540-a28f-446a-bc7f-3b3d143bc351" (UID: "270f7540-a28f-446a-bc7f-3b3d143bc351"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:44.604596 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604584    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-workload-socket" pod "270f7540-a28f-446a-bc7f-3b3d143bc351" (UID: "270f7540-a28f-446a-bc7f-3b3d143bc351"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:44.604872 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604853    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270f7540-a28f-446a-bc7f-3b3d143bc351-istiod-ca-cert" pod "270f7540-a28f-446a-bc7f-3b3d143bc351" (UID: "270f7540-a28f-446a-bc7f-3b3d143bc351"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:44.604936 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604866    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.604936 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604887    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.604936 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604901    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.604936 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.604913    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.606596 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.606570    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-envoy" pod "270f7540-a28f-446a-bc7f-3b3d143bc351" (UID: "270f7540-a28f-446a-bc7f-3b3d143bc351"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:50:44.606859 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.606834    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-podinfo" pod "270f7540-a28f-446a-bc7f-3b3d143bc351" (UID: "270f7540-a28f-446a-bc7f-3b3d143bc351"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:50:44.606967 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.606873    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-token" pod "270f7540-a28f-446a-bc7f-3b3d143bc351" (UID: "270f7540-a28f-446a-bc7f-3b3d143bc351"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:44.606967 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.606899    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270f7540-a28f-446a-bc7f-3b3d143bc351-kube-api-access-k8sqd" pod "270f7540-a28f-446a-bc7f-3b3d143bc351" (UID: "270f7540-a28f-446a-bc7f-3b3d143bc351"). InnerVolumeSpecName "kube-api-access-k8sqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:44.705847 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.705812    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.705847 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.705840    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.705847 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.705849    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270f7540-a28f-446a-bc7f-3b3d143bc351-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.706148 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.705858    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8sqd\" (UniqueName: \"kubernetes.io/projected/270f7540-a28f-446a-bc7f-3b3d143bc351-kube-api-access-k8sqd\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.706148 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.705868    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270f7540-a28f-446a-bc7f-3b3d143bc351-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:44.882393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.882344    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"]
Nov 02 09:50:44.888428 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:44.888402    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8743a5-openshift-default-b59dcb9c-sqnjm"]
Nov 02 09:50:45.316764 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:45.316735    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="270f7540-a28f-446a-bc7f-3b3d143bc351" path="/var/lib/kubelet/pods/270f7540-a28f-446a-bc7f-3b3d143bc351/volumes"
Nov 02 09:50:48.589902 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:48.589871    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"]
Nov 02 09:50:50.244166 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.244138    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"
Nov 02 09:50:50.307728 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.307699    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b9d779-f66b-4283-8eef-d1f80751fef1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b9d779-f66b-4283-8eef-d1f80751fef1-plugins-config-volume\") pod \"29b9d779-f66b-4283-8eef-d1f80751fef1\" (UID: \"29b9d779-f66b-4283-8eef-d1f80751fef1\") "
Nov 02 09:50:50.308237 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.308209    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b9d779-f66b-4283-8eef-d1f80751fef1-plugins-config-volume" pod "29b9d779-f66b-4283-8eef-d1f80751fef1" (UID: "29b9d779-f66b-4283-8eef-d1f80751fef1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:50.411856 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.411027    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b9d779-f66b-4283-8eef-d1f80751fef1-kube-api-access-zc65k\" (UniqueName: \"kubernetes.io/projected/29b9d779-f66b-4283-8eef-d1f80751fef1-kube-api-access-zc65k\") pod \"29b9d779-f66b-4283-8eef-d1f80751fef1\" (UID: \"29b9d779-f66b-4283-8eef-d1f80751fef1\") "
Nov 02 09:50:50.411856 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.411519    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b9d779-f66b-4283-8eef-d1f80751fef1-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:50.441623 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.441542    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b9d779-f66b-4283-8eef-d1f80751fef1-kube-api-access-zc65k" pod "29b9d779-f66b-4283-8eef-d1f80751fef1" (UID: "29b9d779-f66b-4283-8eef-d1f80751fef1"). InnerVolumeSpecName "kube-api-access-zc65k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:50.514019 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.513982    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc65k\" (UniqueName: \"kubernetes.io/projected/29b9d779-f66b-4283-8eef-d1f80751fef1-kube-api-access-zc65k\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:50.607672 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.607632    4614 generic.go:356] "Generic (PLEG): container finished" podID="29b9d779-f66b-4283-8eef-d1f80751fef1" containerID="cc912e937b666b975f1a8c1db05a019241f44868b89f376d9d029b32e0c4f26e" exitCode=0
Nov 02 09:50:50.607840 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.607688    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"
Nov 02 09:50:50.607840 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.607719    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr" event={"ID":"29b9d779-f66b-4283-8eef-d1f80751fef1","Type":"ContainerDied","Data":"cc912e937b666b975f1a8c1db05a019241f44868b89f376d9d029b32e0c4f26e"}
Nov 02 09:50:50.607840 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.607781    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr" event={"ID":"29b9d779-f66b-4283-8eef-d1f80751fef1","Type":"ContainerDied","Data":"c4e65b6edafb3ed34fd6a65a28d96ba2e604cbb3abf4a3305fe462d6f2349f4f"}
Nov 02 09:50:50.607840 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.607810    4614 scope.go:122] "RemoveContainer" containerID="cc912e937b666b975f1a8c1db05a019241f44868b89f376d9d029b32e0c4f26e"
Nov 02 09:50:50.618131 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.618114    4614 scope.go:122] "RemoveContainer" containerID="cc912e937b666b975f1a8c1db05a019241f44868b89f376d9d029b32e0c4f26e"
Nov 02 09:50:50.618392 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:50:50.618371    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc912e937b666b975f1a8c1db05a019241f44868b89f376d9d029b32e0c4f26e\": container with ID starting with cc912e937b666b975f1a8c1db05a019241f44868b89f376d9d029b32e0c4f26e not found: ID does not exist" containerID="cc912e937b666b975f1a8c1db05a019241f44868b89f376d9d029b32e0c4f26e"
Nov 02 09:50:50.618456 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.618401    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc912e937b666b975f1a8c1db05a019241f44868b89f376d9d029b32e0c4f26e"} err="failed to get container status \"cc912e937b666b975f1a8c1db05a019241f44868b89f376d9d029b32e0c4f26e\": rpc error: code = NotFound desc = could not find container \"cc912e937b666b975f1a8c1db05a019241f44868b89f376d9d029b32e0c4f26e\": container with ID starting with cc912e937b666b975f1a8c1db05a019241f44868b89f376d9d029b32e0c4f26e not found: ID does not exist"
Nov 02 09:50:50.632240 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.632212    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"]
Nov 02 09:50:50.638664 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:50.638643    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr"]
Nov 02 09:50:51.225720 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:51.225675    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8743a5-679877bf47-jj7dr" podUID="29b9d779-f66b-4283-8eef-d1f80751fef1" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.64:50052\" within 1s: context deadline exceeded"
Nov 02 09:50:51.225902 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:50:51.225761    4614 logging.go:55] [core] [Channel #257 SubChannel #258]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.64:50052", ServerName: "10.132.0.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.64:50052: operation was canceled"
Nov 02 09:50:51.316262 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:51.316226    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b9d779-f66b-4283-8eef-d1f80751fef1" path="/var/lib/kubelet/pods/29b9d779-f66b-4283-8eef-d1f80751fef1/volumes"
Nov 02 09:50:53.598217 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:53.598168    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp" podUID="cfc66550-db37-4578-a682-3155465834f2" containerName="payload-processing" containerID="cri-o://f96b56ee8cdc8c45ffd5d7bf45a96bd34ee987cd964189a0e21351386b40fd7d" gracePeriod=25
Nov 02 09:50:54.411704 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.411676    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:54.421559 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.421529    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc66550-db37-4578-a682-3155465834f2-kube-api-access-9qd7v\" (UniqueName: \"kubernetes.io/projected/cfc66550-db37-4578-a682-3155465834f2-kube-api-access-9qd7v\") pod \"cfc66550-db37-4578-a682-3155465834f2\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") "
Nov 02 09:50:54.421719 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.421667    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc66550-db37-4578-a682-3155465834f2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cfc66550-db37-4578-a682-3155465834f2-routing-overlay-volume\") pod \"cfc66550-db37-4578-a682-3155465834f2\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") "
Nov 02 09:50:54.421719 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.421695    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc66550-db37-4578-a682-3155465834f2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cfc66550-db37-4578-a682-3155465834f2-provider-credentials\") pod \"cfc66550-db37-4578-a682-3155465834f2\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") "
Nov 02 09:50:54.421804 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.421718    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc66550-db37-4578-a682-3155465834f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc66550-db37-4578-a682-3155465834f2-plugins-config-volume\") pod \"cfc66550-db37-4578-a682-3155465834f2\" (UID: \"cfc66550-db37-4578-a682-3155465834f2\") "
Nov 02 09:50:54.422092 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.422063    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc66550-db37-4578-a682-3155465834f2-routing-overlay-volume" pod "cfc66550-db37-4578-a682-3155465834f2" (UID: "cfc66550-db37-4578-a682-3155465834f2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:54.422177 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.422123    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc66550-db37-4578-a682-3155465834f2-plugins-config-volume" pod "cfc66550-db37-4578-a682-3155465834f2" (UID: "cfc66550-db37-4578-a682-3155465834f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:50:54.423985 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.423960    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc66550-db37-4578-a682-3155465834f2-kube-api-access-9qd7v" pod "cfc66550-db37-4578-a682-3155465834f2" (UID: "cfc66550-db37-4578-a682-3155465834f2"). InnerVolumeSpecName "kube-api-access-9qd7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:54.424130 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.424110    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc66550-db37-4578-a682-3155465834f2-provider-credentials" pod "cfc66550-db37-4578-a682-3155465834f2" (UID: "cfc66550-db37-4578-a682-3155465834f2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:50:54.522743 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.522711    4614 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cfc66550-db37-4578-a682-3155465834f2-routing-overlay-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:54.522743 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.522739    4614 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cfc66550-db37-4578-a682-3155465834f2-provider-credentials\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:54.522743 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.522749    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc66550-db37-4578-a682-3155465834f2-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:54.522984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.522760    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qd7v\" (UniqueName: \"kubernetes.io/projected/cfc66550-db37-4578-a682-3155465834f2-kube-api-access-9qd7v\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:50:54.749749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.749719    4614 generic.go:356] "Generic (PLEG): container finished" podID="cfc66550-db37-4578-a682-3155465834f2" containerID="f96b56ee8cdc8c45ffd5d7bf45a96bd34ee987cd964189a0e21351386b40fd7d" exitCode=0
Nov 02 09:50:54.750135 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.749785    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp" event={"ID":"cfc66550-db37-4578-a682-3155465834f2","Type":"ContainerDied","Data":"f96b56ee8cdc8c45ffd5d7bf45a96bd34ee987cd964189a0e21351386b40fd7d"}
Nov 02 09:50:54.750135 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.749825    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"
Nov 02 09:50:54.750135 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.749868    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp" event={"ID":"cfc66550-db37-4578-a682-3155465834f2","Type":"ContainerDied","Data":"02854b88533dc4b0273fc57c9255caaa432321e0719381c26012c70580ae5da7"}
Nov 02 09:50:54.750135 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.749890    4614 scope.go:122] "RemoveContainer" containerID="f96b56ee8cdc8c45ffd5d7bf45a96bd34ee987cd964189a0e21351386b40fd7d"
Nov 02 09:50:54.761042 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.761019    4614 scope.go:122] "RemoveContainer" containerID="f96b56ee8cdc8c45ffd5d7bf45a96bd34ee987cd964189a0e21351386b40fd7d"
Nov 02 09:50:54.761323 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:50:54.761303    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96b56ee8cdc8c45ffd5d7bf45a96bd34ee987cd964189a0e21351386b40fd7d\": container with ID starting with f96b56ee8cdc8c45ffd5d7bf45a96bd34ee987cd964189a0e21351386b40fd7d not found: ID does not exist" containerID="f96b56ee8cdc8c45ffd5d7bf45a96bd34ee987cd964189a0e21351386b40fd7d"
Nov 02 09:50:54.761414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.761333    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96b56ee8cdc8c45ffd5d7bf45a96bd34ee987cd964189a0e21351386b40fd7d"} err="failed to get container status \"f96b56ee8cdc8c45ffd5d7bf45a96bd34ee987cd964189a0e21351386b40fd7d\": rpc error: code = NotFound desc = could not find container \"f96b56ee8cdc8c45ffd5d7bf45a96bd34ee987cd964189a0e21351386b40fd7d\": container with ID starting with f96b56ee8cdc8c45ffd5d7bf45a96bd34ee987cd964189a0e21351386b40fd7d not found: ID does not exist"
Nov 02 09:50:54.774917 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.774893    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"]
Nov 02 09:50:54.781627 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:54.781599    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5db5crcsp"]
Nov 02 09:50:55.026631 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:55.026559    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"]
Nov 02 09:50:55.030171 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:55.030143    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4d50b8-dvbj5"]
Nov 02 09:50:55.316329 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:55.316252    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12148caa-3e3f-471d-a0f8-572e58e35b7b" path="/var/lib/kubelet/pods/12148caa-3e3f-471d-a0f8-572e58e35b7b/volumes"
Nov 02 09:50:55.316710 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:55.316696    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc66550-db37-4578-a682-3155465834f2" path="/var/lib/kubelet/pods/cfc66550-db37-4578-a682-3155465834f2/volumes"
Nov 02 09:50:59.005279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.005242    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"]
Nov 02 09:50:59.006050 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.005991    4614 cpu_manager.go:414] "containerMap: removing container" podUID="a5ed6879-0ce6-4705-a1bc-eefbfef9157f" containerName="istio-proxy"
Nov 02 09:50:59.006184 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.006065    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ed6879-0ce6-4705-a1bc-eefbfef9157f" containerName="istio-proxy"
Nov 02 09:50:59.006184 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.006105    4614 cpu_manager.go:414] "containerMap: removing container" podUID="cfc66550-db37-4578-a682-3155465834f2" containerName="payload-processing"
Nov 02 09:50:59.006184 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.006114    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc66550-db37-4578-a682-3155465834f2" containerName="payload-processing"
Nov 02 09:50:59.006184 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.006130    4614 cpu_manager.go:414] "containerMap: removing container" podUID="29b9d779-f66b-4283-8eef-d1f80751fef1" containerName="payload-pre-processing"
Nov 02 09:50:59.006184 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.006138    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b9d779-f66b-4283-8eef-d1f80751fef1" containerName="payload-pre-processing"
Nov 02 09:50:59.006470 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.006203    4614 cpu_manager.go:414] "containerMap: removing container" podUID="270f7540-a28f-446a-bc7f-3b3d143bc351" containerName="istio-proxy"
Nov 02 09:50:59.006470 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.006210    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="270f7540-a28f-446a-bc7f-3b3d143bc351" containerName="istio-proxy"
Nov 02 09:50:59.007022 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.006999    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="270f7540-a28f-446a-bc7f-3b3d143bc351" containerName="istio-proxy"
Nov 02 09:50:59.007181 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.007027    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ed6879-0ce6-4705-a1bc-eefbfef9157f" containerName="istio-proxy"
Nov 02 09:50:59.007181 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.007039    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc66550-db37-4578-a682-3155465834f2" containerName="payload-processing"
Nov 02 09:50:59.007181 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.007049    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b9d779-f66b-4283-8eef-d1f80751fef1" containerName="payload-pre-processing"
Nov 02 09:50:59.013497 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.013462    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"]
Nov 02 09:50:59.013856 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.013833    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:50:59.042334 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.042311    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-712810ea-serving-cert\""
Nov 02 09:50:59.101656 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.101621    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d274b66a-af35-4a0b-ac88-a4aa419358b4-openshift-service-ca\") pod \"maas-api-e2e-ait-712810ea-69656bb45d-z5zfj\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:50:59.101656 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.101657    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d274b66a-af35-4a0b-ac88-a4aa419358b4-maas-api-tls\") pod \"maas-api-e2e-ait-712810ea-69656bb45d-z5zfj\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:50:59.101916 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.101856    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c45jv\" (UniqueName: \"kubernetes.io/projected/d274b66a-af35-4a0b-ac88-a4aa419358b4-kube-api-access-c45jv\") pod \"maas-api-e2e-ait-712810ea-69656bb45d-z5zfj\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:50:59.101916 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.101906    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d274b66a-af35-4a0b-ac88-a4aa419358b4-metrics-certs\") pod \"maas-api-e2e-ait-712810ea-69656bb45d-z5zfj\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:50:59.221408 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.221141    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c45jv\" (UniqueName: \"kubernetes.io/projected/d274b66a-af35-4a0b-ac88-a4aa419358b4-kube-api-access-c45jv\") pod \"maas-api-e2e-ait-712810ea-69656bb45d-z5zfj\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:50:59.221408 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.221200    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d274b66a-af35-4a0b-ac88-a4aa419358b4-metrics-certs\") pod \"maas-api-e2e-ait-712810ea-69656bb45d-z5zfj\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:50:59.221408 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.221376    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d274b66a-af35-4a0b-ac88-a4aa419358b4-openshift-service-ca\") pod \"maas-api-e2e-ait-712810ea-69656bb45d-z5zfj\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:50:59.221408 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.221414    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d274b66a-af35-4a0b-ac88-a4aa419358b4-maas-api-tls\") pod \"maas-api-e2e-ait-712810ea-69656bb45d-z5zfj\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:50:59.235453 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.223495    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d274b66a-af35-4a0b-ac88-a4aa419358b4-openshift-service-ca\") pod \"maas-api-e2e-ait-712810ea-69656bb45d-z5zfj\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:50:59.245087 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.241942    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d274b66a-af35-4a0b-ac88-a4aa419358b4-maas-api-tls\") pod \"maas-api-e2e-ait-712810ea-69656bb45d-z5zfj\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:50:59.245087 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.242570    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d274b66a-af35-4a0b-ac88-a4aa419358b4-metrics-certs\") pod \"maas-api-e2e-ait-712810ea-69656bb45d-z5zfj\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:50:59.263246 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.263150    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c45jv\" (UniqueName: \"kubernetes.io/projected/d274b66a-af35-4a0b-ac88-a4aa419358b4-kube-api-access-c45jv\") pod \"maas-api-e2e-ait-712810ea-69656bb45d-z5zfj\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:50:59.330694 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.330661    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:50:59.615902 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:50:59.615865    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd274b66a_af35_4a0b_ac88_a4aa419358b4.slice/crio-55ff53c0ef87db48288715ba2b37898c77509592bd7d300863061a7ca0efb381 WatchSource:0}: Error finding container 55ff53c0ef87db48288715ba2b37898c77509592bd7d300863061a7ca0efb381: Status 404 returned error can't find the container with id 55ff53c0ef87db48288715ba2b37898c77509592bd7d300863061a7ca0efb381
Nov 02 09:50:59.616788 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.616760    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"]
Nov 02 09:50:59.809563 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:50:59.809440    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj" event={"ID":"d274b66a-af35-4a0b-ac88-a4aa419358b4","Type":"ContainerStarted","Data":"55ff53c0ef87db48288715ba2b37898c77509592bd7d300863061a7ca0efb381"}
Nov 02 09:51:00.846455 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:00.832143    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj" event={"ID":"d274b66a-af35-4a0b-ac88-a4aa419358b4","Type":"ContainerStarted","Data":"34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c"}
Nov 02 09:51:00.846455 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:00.833217    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:51:07.861395 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:07.861334    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:51:07.883799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:07.883746    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj" podStartSLOduration=9.258798185 podStartE2EDuration="9.883732652s" totalImagesPullingTime="624.934467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:00.862605378 +0000 UTC m=+941.695263458" watchObservedRunningTime="2026-11-02 09:51:07.883732652 +0000 UTC m=+948.716390731"
Nov 02 09:51:08.042738 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:08.042157    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"]
Nov 02 09:51:08.049586 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:08.048648    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdcc2d17-kh9hw"]
Nov 02 09:51:09.317148 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:09.317113    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2647b34f-b117-4685-82c2-61eeeca4837e" path="/var/lib/kubelet/pods/2647b34f-b117-4685-82c2-61eeeca4837e/volumes"
Nov 02 09:51:09.639072 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:51:09.638992    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:51:14.812376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:14.811487    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:15.119591 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:15.119492    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:15.356381 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:15.346457    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:15.510378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:15.509873    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:15.666378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:15.659380    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:15.693501 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:15.693450    4614 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:15.840878 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:15.840776    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:15.843169 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:15.843139    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:51:15.869623 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:15.869502    4614 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:15.977529 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:15.977480    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 02 09:51:16.102470 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:16.072465    4614 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:16.120162 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:16.120118    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:16.121889 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:16.121864    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:51:16.212592 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:16.212546    4614 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:16.241012 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:16.240984    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:51:16.354587 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:16.354488    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 02 09:51:16.359490 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:16.359460    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:51:16.548399 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:16.510621    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:16.548399 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:16.512497    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:51:16.845499 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:16.845452    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:16.993703 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:16.993185    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 02 09:51:16.995742 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:16.994319    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:51:17.364183 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:17.363439    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 02 09:51:17.381389 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:17.370317    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:51:17.381389 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:17.370433    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:17.477402 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:17.460633    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:51:17.523374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:17.523313    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Nov 02 09:51:17.655473 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:17.655388    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:51:17.738943 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:17.738914    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:51:17.833543 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:17.833511    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:51:17.833815 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:17.833782    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:17.858599 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:17.858573    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:51:18.280368 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.280331    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"]
Nov 02 09:51:18.284556 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.284534    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.298123 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.298099    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"]
Nov 02 09:51:18.328279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.328238    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7bwg\" (UniqueName: \"kubernetes.io/projected/125f651d-4e3c-4f0e-8748-50e55a48407d-kube-api-access-r7bwg\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.328491 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.328420    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.328491 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.328447    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.328491 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.328480    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.328681 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.328615    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/125f651d-4e3c-4f0e-8748-50e55a48407d-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.328681 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.328655    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.328787 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.328744    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.328787 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.328773    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.328901 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.328815    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.438534 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.438487    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.438727 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.438541    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.438727 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.438613    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.438727 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.438674    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/125f651d-4e3c-4f0e-8748-50e55a48407d-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.438727 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.438710    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.438951 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.438797    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.438951 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.438824    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.438951 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.438916    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.439114 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.439023    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7bwg\" (UniqueName: \"kubernetes.io/projected/125f651d-4e3c-4f0e-8748-50e55a48407d-kube-api-access-r7bwg\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.441961 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.441924    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.442452 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.442428    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.443134 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.443100    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.443676 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.443387    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/125f651d-4e3c-4f0e-8748-50e55a48407d-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.444018 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.443987    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.452399 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.452378    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.454058 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.454037    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.455545 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.455520    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7bwg\" (UniqueName: \"kubernetes.io/projected/125f651d-4e3c-4f0e-8748-50e55a48407d-kube-api-access-r7bwg\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.459099 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.459076    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.600859 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.600770    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:18.916858 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.916753    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"]
Nov 02 09:51:18.980643 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.980587    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:51:18.981005 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.980964    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:51:18.981200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:18.981163    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:51:19.179867 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:19.179785    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl" event={"ID":"125f651d-4e3c-4f0e-8748-50e55a48407d","Type":"ContainerStarted","Data":"8cfbc7c79cdaa827a0fa5bce605837560935f7a6bdfcdaa4b00e087aa22bd795"}
Nov 02 09:51:19.179867 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:19.179856    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl" event={"ID":"125f651d-4e3c-4f0e-8748-50e55a48407d","Type":"ContainerStarted","Data":"9a7cec6b20f2bc1e1d35c2553b9783593cd0a733b6fc1905f07d220b2992dd51"}
Nov 02 09:51:19.208146 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:19.208082    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl" podStartSLOduration=1.208061852 podStartE2EDuration="1.208061852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:19.202905809 +0000 UTC m=+960.035563890" watchObservedRunningTime="2026-11-02 09:51:19.208061852 +0000 UTC m=+960.040719933"
Nov 02 09:51:19.601131 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:19.601081    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:20.602646 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:20.602610    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 02 09:51:20.603065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:20.602668    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl" podUID="125f651d-4e3c-4f0e-8748-50e55a48407d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 02 09:51:21.603289 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:21.603251    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 02 09:51:21.603705 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:21.603305    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl" podUID="125f651d-4e3c-4f0e-8748-50e55a48407d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 02 09:51:22.601256 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:22.601219    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 02 09:51:22.601470 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:22.601300    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl" podUID="125f651d-4e3c-4f0e-8748-50e55a48407d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 02 09:51:22.702152 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:22.702117    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"]
Nov 02 09:51:22.709573 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:22.709545    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:22.719321 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:22.719294    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"]
Nov 02 09:51:22.914187 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:22.914103    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:22.914187 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:22.914163    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:22.914393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:22.914256    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:22.914393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:22.914327    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:22.914393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:22.914390    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:22.914501 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:22.914423    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:22.914501 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:22.914443    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:22.914567 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:22.914547    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwd4s\" (UniqueName: \"kubernetes.io/projected/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-kube-api-access-dwd4s\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:22.914603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:22.914567    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.015251 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.015214    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.015251 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.015268    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.015532 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.015315    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.015532 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.015346    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.015532 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.015431    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd4s\" (UniqueName: \"kubernetes.io/projected/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-kube-api-access-dwd4s\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.015532 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.015458    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.015532 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.015529    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.015791 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.015582    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.015791 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.015621    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.015791 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.015740    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.016078 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.016053    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.016285 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.016255    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.016605 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.016569    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.016846 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.016630    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.018178 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.018152    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.018704 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.018678    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.052774 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.052747    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.053745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.053722    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwd4s\" (UniqueName: \"kubernetes.io/projected/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-kube-api-access-dwd4s\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.068169 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.068143    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:23.366273 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.366240    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"]
Nov 02 09:51:23.367421 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:51:23.367398    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6421f1c9_7fdd_4cb1_8313_6c9191cb69f7.slice/crio-86f1eda99e4a796b89a4a52003db419fa4e8a64d6ceda03b1b2e453749e95f6e WatchSource:0}: Error finding container 86f1eda99e4a796b89a4a52003db419fa4e8a64d6ceda03b1b2e453749e95f6e: Status 404 returned error can't find the container with id 86f1eda99e4a796b89a4a52003db419fa4e8a64d6ceda03b1b2e453749e95f6e
Nov 02 09:51:23.370049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.370014    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:51:23.370171 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.370093    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:51:23.370171 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.370137    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:51:23.610987 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.610746    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 02 09:51:23.610987 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:23.610801    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl" podUID="125f651d-4e3c-4f0e-8748-50e55a48407d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 02 09:51:24.213182 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:24.213147    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" event={"ID":"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7","Type":"ContainerStarted","Data":"74df00d7c2e5d4f77f192acfb3f666a5e263dd68b9be638a07a7b1434646ab10"}
Nov 02 09:51:24.213585 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:24.213202    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" event={"ID":"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7","Type":"ContainerStarted","Data":"86f1eda99e4a796b89a4a52003db419fa4e8a64d6ceda03b1b2e453749e95f6e"}
Nov 02 09:51:24.601958 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:24.601920    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 02 09:51:24.602146 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:24.601976    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl" podUID="125f651d-4e3c-4f0e-8748-50e55a48407d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 02 09:51:25.068982 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:25.068935    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:25.070832 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:25.070806    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 09:51:25.070935 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:25.070863    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" podUID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 09:51:25.924619 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:25.924588    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:25.925714 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:25.925684    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:25.931794 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:25.931771    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 09:51:25.961670 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:25.961594    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" podStartSLOduration=3.961575336 podStartE2EDuration="3.961575336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:24.255414383 +0000 UTC m=+965.088072482" watchObservedRunningTime="2026-11-02 09:51:25.961575336 +0000 UTC m=+966.794233416"
Nov 02 09:51:26.056690 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:26.056656    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"]
Nov 02 09:51:26.058511 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:26.058478    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93664871-9gmj5"]
Nov 02 09:51:26.069018 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:26.068989    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 09:51:26.069132 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:26.069037    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" podUID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 09:51:27.071407 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:27.070860    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 09:51:27.071407 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:27.070945    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" podUID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 09:51:27.357434 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:27.349040    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d51871fc-4acd-40e7-b1fd-c012e2c5cebe" path="/var/lib/kubelet/pods/d51871fc-4acd-40e7-b1fd-c012e2c5cebe/volumes"
Nov 02 09:51:28.074015 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:28.069068    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 09:51:28.074015 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:28.069129    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" podUID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 09:51:29.078205 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:29.071744    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 09:51:29.078205 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:29.071810    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" podUID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 09:51:29.282381 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:29.254443    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:29.476742 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:29.475709    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 02 09:51:29.868429 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:29.867602    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:30.120908 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:30.120453    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 09:51:30.120908 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:30.120509    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" podUID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 09:51:30.288047 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:30.284368    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:30.288047 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:30.285041    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:51:30.504597 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:30.504550    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:30.524376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:30.510124    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:51:30.663431 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:30.644727    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 02 09:51:30.736599 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:30.736545    4614 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:30.737850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:30.737817    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 02 09:51:30.893978 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:30.893870    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 09:51:30.903465 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:30.903412    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Nov 02 09:51:30.932888 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:30.932855    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:51:31.188664 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:31.188546    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 09:51:31.210383 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:31.205797    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" podUID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 09:51:31.345485 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:31.334704    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:31.528392 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:31.519449    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:31.660385 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:31.653446    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 02 09:51:31.660385 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:31.654284    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:51:31.786434 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:31.785321    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:31.786434 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:31.786130    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:51:31.942380 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:31.906639    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:32.100477 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:31.998934    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:32.100477 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:31.999822    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:51:32.100477 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:32.070684    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 09:51:32.100477 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:32.070737    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" podUID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 09:51:32.244004 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:32.243468    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:51:32.244004 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:32.243582    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492" podUID="ae2706aa-d41a-463a-acf6-79c6b2705f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:32.405044 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:32.354463    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:51:32.441735 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:32.410643    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:32.570544 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:32.570507    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:51:32.718441 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:32.691520    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:51:32.718441 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:32.707431    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:51:32.718774 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:32.718739    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 02 09:51:32.754666 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:32.754637    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:51:33.122115 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:33.122034    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 02 09:51:33.122506 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:33.122474    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" podUID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 02 09:51:33.223673 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:33.223643    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:51:33.297377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:33.296534    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 09:51:34.073760 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:34.073727    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:34.074324 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:34.074301    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:34.075119 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:34.075098    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:51:36.702073 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:36.702030    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"]
Nov 02 09:51:36.702605 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:36.702288    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj" podUID="d274b66a-af35-4a0b-ac88-a4aa419358b4" containerName="maas-api" containerID="cri-o://34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c" gracePeriod=30
Nov 02 09:51:36.806057 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:51:36.806014    4614 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd274b66a_af35_4a0b_ac88_a4aa419358b4.slice/crio-conmon-34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd274b66a_af35_4a0b_ac88_a4aa419358b4.slice/crio-34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 09:51:37.188163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.188131    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:51:37.297986 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.297951    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d274b66a-af35-4a0b-ac88-a4aa419358b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d274b66a-af35-4a0b-ac88-a4aa419358b4-metrics-certs\") pod \"d274b66a-af35-4a0b-ac88-a4aa419358b4\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") "
Nov 02 09:51:37.298171 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.298154    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d274b66a-af35-4a0b-ac88-a4aa419358b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d274b66a-af35-4a0b-ac88-a4aa419358b4-maas-api-tls\") pod \"d274b66a-af35-4a0b-ac88-a4aa419358b4\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") "
Nov 02 09:51:37.298218 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.298200    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d274b66a-af35-4a0b-ac88-a4aa419358b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d274b66a-af35-4a0b-ac88-a4aa419358b4-openshift-service-ca\") pod \"d274b66a-af35-4a0b-ac88-a4aa419358b4\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") "
Nov 02 09:51:37.298275 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.298261    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d274b66a-af35-4a0b-ac88-a4aa419358b4-kube-api-access-c45jv\" (UniqueName: \"kubernetes.io/projected/d274b66a-af35-4a0b-ac88-a4aa419358b4-kube-api-access-c45jv\") pod \"d274b66a-af35-4a0b-ac88-a4aa419358b4\" (UID: \"d274b66a-af35-4a0b-ac88-a4aa419358b4\") "
Nov 02 09:51:37.298676 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.298647    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d274b66a-af35-4a0b-ac88-a4aa419358b4-openshift-service-ca" pod "d274b66a-af35-4a0b-ac88-a4aa419358b4" (UID: "d274b66a-af35-4a0b-ac88-a4aa419358b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:51:37.300159 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.300129    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d274b66a-af35-4a0b-ac88-a4aa419358b4-metrics-certs" pod "d274b66a-af35-4a0b-ac88-a4aa419358b4" (UID: "d274b66a-af35-4a0b-ac88-a4aa419358b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:51:37.300280 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.300264    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d274b66a-af35-4a0b-ac88-a4aa419358b4-maas-api-tls" pod "d274b66a-af35-4a0b-ac88-a4aa419358b4" (UID: "d274b66a-af35-4a0b-ac88-a4aa419358b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:51:37.300405 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.300383    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d274b66a-af35-4a0b-ac88-a4aa419358b4-kube-api-access-c45jv" pod "d274b66a-af35-4a0b-ac88-a4aa419358b4" (UID: "d274b66a-af35-4a0b-ac88-a4aa419358b4"). InnerVolumeSpecName "kube-api-access-c45jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:51:37.399696 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.399665    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d274b66a-af35-4a0b-ac88-a4aa419358b4-maas-api-tls\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:37.399696 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.399696    4614 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d274b66a-af35-4a0b-ac88-a4aa419358b4-openshift-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:37.399899 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.399706    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c45jv\" (UniqueName: \"kubernetes.io/projected/d274b66a-af35-4a0b-ac88-a4aa419358b4-kube-api-access-c45jv\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:37.399899 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.399715    4614 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d274b66a-af35-4a0b-ac88-a4aa419358b4-metrics-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:51:37.505845 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.505813    4614 generic.go:356] "Generic (PLEG): container finished" podID="d274b66a-af35-4a0b-ac88-a4aa419358b4" containerID="34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c" exitCode=0
Nov 02 09:51:37.506026 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.505859    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"
Nov 02 09:51:37.506026 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.505872    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj" event={"ID":"d274b66a-af35-4a0b-ac88-a4aa419358b4","Type":"ContainerDied","Data":"34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c"}
Nov 02 09:51:37.506026 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.505911    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj" event={"ID":"d274b66a-af35-4a0b-ac88-a4aa419358b4","Type":"ContainerDied","Data":"55ff53c0ef87db48288715ba2b37898c77509592bd7d300863061a7ca0efb381"}
Nov 02 09:51:37.506026 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.505933    4614 scope.go:122] "RemoveContainer" containerID="34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c"
Nov 02 09:51:37.515768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.515751    4614 scope.go:122] "RemoveContainer" containerID="34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c"
Nov 02 09:51:37.516026 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:51:37.516005    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c\": container with ID starting with 34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c not found: ID does not exist" containerID="34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c"
Nov 02 09:51:37.516070 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.516036    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c"} err="failed to get container status \"34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c\": rpc error: code = NotFound desc = could not find container \"34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c\": container with ID starting with 34841b86a3646e355fa1065ecd5285efa8529e2194c117256499fd43c25e451c not found: ID does not exist"
Nov 02 09:51:37.528206 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.528182    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"]
Nov 02 09:51:37.532830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:37.532804    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-712810ea-69656bb45d-z5zfj"]
Nov 02 09:51:39.415378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:39.410997    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d274b66a-af35-4a0b-ac88-a4aa419358b4" path="/var/lib/kubelet/pods/d274b66a-af35-4a0b-ac88-a4aa419358b4/volumes"
Nov 02 09:51:39.639780 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:51:39.639744    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:51:43.318784 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.318750    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"]
Nov 02 09:51:43.319540 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.319520    4614 cpu_manager.go:414] "containerMap: removing container" podUID="d274b66a-af35-4a0b-ac88-a4aa419358b4" containerName="maas-api"
Nov 02 09:51:43.319583 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.319544    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d274b66a-af35-4a0b-ac88-a4aa419358b4" containerName="maas-api"
Nov 02 09:51:43.319755 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.319741    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="d274b66a-af35-4a0b-ac88-a4aa419358b4" containerName="maas-api"
Nov 02 09:51:43.325189 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.325165    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"
Nov 02 09:51:43.327106 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.327079    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"]
Nov 02 09:51:43.406824 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.406763    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4c342351-dockercfg-zd9gm\""
Nov 02 09:51:43.442375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.418586    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4c342351\""
Nov 02 09:51:43.443996 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.443474    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s48qw\" (UniqueName: \"kubernetes.io/projected/847784f5-ed03-4ca8-9450-6546dbee4e8f-kube-api-access-s48qw\") pod \"payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n\" (UID: \"847784f5-ed03-4ca8-9450-6546dbee4e8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"
Nov 02 09:51:43.444580 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.444544    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/847784f5-ed03-4ca8-9450-6546dbee4e8f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n\" (UID: \"847784f5-ed03-4ca8-9450-6546dbee4e8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"
Nov 02 09:51:43.554374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.552402    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/847784f5-ed03-4ca8-9450-6546dbee4e8f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n\" (UID: \"847784f5-ed03-4ca8-9450-6546dbee4e8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"
Nov 02 09:51:43.554374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.552512    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s48qw\" (UniqueName: \"kubernetes.io/projected/847784f5-ed03-4ca8-9450-6546dbee4e8f-kube-api-access-s48qw\") pod \"payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n\" (UID: \"847784f5-ed03-4ca8-9450-6546dbee4e8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"
Nov 02 09:51:43.554374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.553844    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/847784f5-ed03-4ca8-9450-6546dbee4e8f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n\" (UID: \"847784f5-ed03-4ca8-9450-6546dbee4e8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"
Nov 02 09:51:43.567597 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.567570    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s48qw\" (UniqueName: \"kubernetes.io/projected/847784f5-ed03-4ca8-9450-6546dbee4e8f-kube-api-access-s48qw\") pod \"payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n\" (UID: \"847784f5-ed03-4ca8-9450-6546dbee4e8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"
Nov 02 09:51:43.641971 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.641884    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"
Nov 02 09:51:43.924324 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:43.924295    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"]
Nov 02 09:51:43.924943 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:51:43.924915    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847784f5_ed03_4ca8_9450_6546dbee4e8f.slice/crio-61691505a0e1fa76e34ebd2813b300bab1d42514ce05fba781a3e4f6b6e87a53 WatchSource:0}: Error finding container 61691505a0e1fa76e34ebd2813b300bab1d42514ce05fba781a3e4f6b6e87a53: Status 404 returned error can't find the container with id 61691505a0e1fa76e34ebd2813b300bab1d42514ce05fba781a3e4f6b6e87a53
Nov 02 09:51:44.755361 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:44.755319    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n" event={"ID":"847784f5-ed03-4ca8-9450-6546dbee4e8f","Type":"ContainerStarted","Data":"99768c24418c9afb29a4d6c904edac6ff14b36d68eee1e7c4e2f4a623420414c"}
Nov 02 09:51:44.755846 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:44.755608    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"
Nov 02 09:51:44.755846 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:44.755678    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n" event={"ID":"847784f5-ed03-4ca8-9450-6546dbee4e8f","Type":"ContainerStarted","Data":"61691505a0e1fa76e34ebd2813b300bab1d42514ce05fba781a3e4f6b6e87a53"}
Nov 02 09:51:44.778141 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:44.778089    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n" podStartSLOduration=1.778077181 podStartE2EDuration="1.778077181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:44.776110986 +0000 UTC m=+985.608769068" watchObservedRunningTime="2026-11-02 09:51:44.778077181 +0000 UTC m=+985.610735263"
Nov 02 09:51:50.103862 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.103823    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"]
Nov 02 09:51:50.107898 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.107876    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:50.118527 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.118499    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4c342351-serving-cert\""
Nov 02 09:51:50.123861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.123834    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"]
Nov 02 09:51:50.230186 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.230142    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f4gf\" (UniqueName: \"kubernetes.io/projected/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-kube-api-access-6f4gf\") pod \"maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:50.237167 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.237133    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-metrics-certs\") pod \"maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:50.244650 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.244605    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-openshift-service-ca\") pod \"maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:50.245048 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.245025    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-maas-api-tls\") pod \"maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:50.346066 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.346030    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-openshift-service-ca\") pod \"maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:50.346257 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.346141    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-maas-api-tls\") pod \"maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:50.346257 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.346191    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f4gf\" (UniqueName: \"kubernetes.io/projected/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-kube-api-access-6f4gf\") pod \"maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:50.346444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.346256    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-metrics-certs\") pod \"maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:50.346819 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.346793    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-openshift-service-ca\") pod \"maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:50.348764 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.348740    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-maas-api-tls\") pod \"maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:50.348848 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.348770    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-metrics-certs\") pod \"maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:50.358619 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.358552    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f4gf\" (UniqueName: \"kubernetes.io/projected/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-kube-api-access-6f4gf\") pod \"maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:50.427835 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.427799    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:50.792963 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:50.792931    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"]
Nov 02 09:51:51.804590 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:51.804552    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph" event={"ID":"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae","Type":"ContainerStarted","Data":"319e6ce960e77e8d06e5b48572996e49349f8ab7ace639abeed80f84d900e090"}
Nov 02 09:51:51.805046 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:51.804612    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph" event={"ID":"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae","Type":"ContainerStarted","Data":"f131a04f4cf444c5e1e93960829dfbbf4f8de26e8376e29071a671edb555826b"}
Nov 02 09:51:51.805046 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:51.804639    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:51:51.833304 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:51.833252    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph" podStartSLOduration=1.399720864 podStartE2EDuration="1.833238084s" totalImagesPullingTime="433.51722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:51:51.830065754 +0000 UTC m=+992.662723833" watchObservedRunningTime="2026-11-02 09:51:51.833238084 +0000 UTC m=+992.665896164"
Nov 02 09:51:54.242467 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:54.242415    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"]
Nov 02 09:51:54.313379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:54.257996    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0188acf8-wkhq9"]
Nov 02 09:51:55.442374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:55.442172    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a0587e-2931-40a4-a73b-be4a43a40030" path="/var/lib/kubelet/pods/94a0587e-2931-40a4-a73b-be4a43a40030/volumes"
Nov 02 09:51:55.772377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:55.771524    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"
Nov 02 09:51:57.846949 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:51:57.846921    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:52:03.372848 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.372807    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"]
Nov 02 09:52:03.378536 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.378507    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.391447 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.391416    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"]
Nov 02 09:52:03.507602 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.507566    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.507763 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.507621    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.507763 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.507716    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rr4\" (UniqueName: \"kubernetes.io/projected/adde7f6c-619c-4d32-9d5e-0540c56c96fc-kube-api-access-52rr4\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.507845 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.507765    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.507845 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.507784    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.507845 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.507801    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.507845 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.507818    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.507845 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.507835    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.508040 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.508026    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.609294 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.609260    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.609498 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.609302    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.609498 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.609321    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.609498 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.609340    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.609498 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.609394    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.609745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.609504    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.609745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.609594    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.609745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.609677    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.609745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.609726    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rr4\" (UniqueName: \"kubernetes.io/projected/adde7f6c-619c-4d32-9d5e-0540c56c96fc-kube-api-access-52rr4\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.609939 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.609751    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.609939 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.609806    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.609939 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.609825    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.609939 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.609850    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.610434 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.610411    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.611962 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.611938    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.612226 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.612203    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.622146 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.622103    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rr4\" (UniqueName: \"kubernetes.io/projected/adde7f6c-619c-4d32-9d5e-0540c56c96fc-kube-api-access-52rr4\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.622751 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.622727    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:03.699508 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:03.699436    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:04.059503 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:04.059474    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"]
Nov 02 09:52:04.060339 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:52:04.060312    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadde7f6c_619c_4d32_9d5e_0540c56c96fc.slice/crio-54b3c1aff9cc6ba056ad33ec3272532e6d63ac1cff4595950b7c1b366b8b9c62 WatchSource:0}: Error finding container 54b3c1aff9cc6ba056ad33ec3272532e6d63ac1cff4595950b7c1b366b8b9c62: Status 404 returned error can't find the container with id 54b3c1aff9cc6ba056ad33ec3272532e6d63ac1cff4595950b7c1b366b8b9c62
Nov 02 09:52:04.062911 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:04.062876    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:52:04.063030 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:04.062954    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:52:04.063030 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:04.063000    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:52:04.150966 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:04.150932    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg" event={"ID":"adde7f6c-619c-4d32-9d5e-0540c56c96fc","Type":"ContainerStarted","Data":"3cc6d4cb080599d75446c77f1d0d3314d926b0eaa40e182e42d03481f4e69424"}
Nov 02 09:52:04.151126 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:04.150999    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg" event={"ID":"adde7f6c-619c-4d32-9d5e-0540c56c96fc","Type":"ContainerStarted","Data":"54b3c1aff9cc6ba056ad33ec3272532e6d63ac1cff4595950b7c1b366b8b9c62"}
Nov 02 09:52:04.178837 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:04.178771    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg" podStartSLOduration=1.178750032 podStartE2EDuration="1.178750032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:04.173063787 +0000 UTC m=+1005.005721868" watchObservedRunningTime="2026-11-02 09:52:04.178750032 +0000 UTC m=+1005.011408114"
Nov 02 09:52:04.699555 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:04.699517    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:05.037376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:05.026122    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"]
Nov 02 09:52:05.056291 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:05.056256    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4d2766-jzkmh"]
Nov 02 09:52:05.316803 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:05.316722    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a614f25-277c-4485-9c5e-93630dd5bb2f" path="/var/lib/kubelet/pods/4a614f25-277c-4485-9c5e-93630dd5bb2f/volumes"
Nov 02 09:52:05.701673 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:05.701585    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 09:52:05.702033 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:05.701661    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg" podUID="adde7f6c-619c-4d32-9d5e-0540c56c96fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 09:52:06.700189 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:06.700151    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 09:52:06.700404 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:06.700234    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg" podUID="adde7f6c-619c-4d32-9d5e-0540c56c96fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 09:52:07.699788 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:07.699753    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 09:52:07.700181 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:07.699825    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg" podUID="adde7f6c-619c-4d32-9d5e-0540c56c96fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 09:52:08.700644 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:08.700605    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 09:52:08.701046 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:08.700660    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg" podUID="adde7f6c-619c-4d32-9d5e-0540c56c96fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 09:52:09.658796 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:52:09.658752    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:52:09.701971 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:09.701938    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 09:52:09.702620 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:09.702589    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg" podUID="adde7f6c-619c-4d32-9d5e-0540c56c96fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 09:52:10.053564 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:10.053522    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"]
Nov 02 09:52:10.057414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:10.056994    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b22726a-sw7kx"]
Nov 02 09:52:10.699797 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:10.699752    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 02 09:52:10.699997 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:10.699828    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg" podUID="adde7f6c-619c-4d32-9d5e-0540c56c96fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 02 09:52:11.317073 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:11.317041    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916f8630-c2f0-431f-92d2-0a677bb0921a" path="/var/lib/kubelet/pods/916f8630-c2f0-431f-92d2-0a677bb0921a/volumes"
Nov 02 09:52:11.706273 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:11.706239    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:11.706701 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:11.706661    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:11.707607 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:11.707588    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 09:52:12.459960 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:12.459923    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"]
Nov 02 09:52:14.656536 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:14.654082    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"]
Nov 02 09:52:14.656536 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:14.654328    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph" podUID="e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae" containerName="maas-api" containerID="cri-o://319e6ce960e77e8d06e5b48572996e49349f8ab7ace639abeed80f84d900e090" gracePeriod=30
Nov 02 09:52:15.435791 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:15.435752    4614 generic.go:356] "Generic (PLEG): container finished" podID="e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae" containerID="319e6ce960e77e8d06e5b48572996e49349f8ab7ace639abeed80f84d900e090" exitCode=0
Nov 02 09:52:15.436392 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:15.436342    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph" event={"ID":"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae","Type":"ContainerDied","Data":"319e6ce960e77e8d06e5b48572996e49349f8ab7ace639abeed80f84d900e090"}
Nov 02 09:52:16.432376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.431516    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:52:16.589427 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.587686    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-metrics-certs\") pod \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") "
Nov 02 09:52:16.589427 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.587750    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-kube-api-access-6f4gf\" (UniqueName: \"kubernetes.io/projected/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-kube-api-access-6f4gf\") pod \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") "
Nov 02 09:52:16.589427 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.587803    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-maas-api-tls\") pod \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") "
Nov 02 09:52:16.589427 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.587987    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-openshift-service-ca\") pod \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\" (UID: \"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae\") "
Nov 02 09:52:16.610377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.592559    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-openshift-service-ca" pod "e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae" (UID: "e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:52:16.692423 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.664762    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph" event={"ID":"e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae","Type":"ContainerDied","Data":"f131a04f4cf444c5e1e93960829dfbbf4f8de26e8376e29071a671edb555826b"}
Nov 02 09:52:16.692423 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.664838    4614 scope.go:122] "RemoveContainer" containerID="319e6ce960e77e8d06e5b48572996e49349f8ab7ace639abeed80f84d900e090"
Nov 02 09:52:16.692423 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.665019    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"
Nov 02 09:52:16.692423 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.666534    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-kube-api-access-6f4gf" pod "e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae" (UID: "e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae"). InnerVolumeSpecName "kube-api-access-6f4gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:16.692423 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.666607    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-maas-api-tls" pod "e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae" (UID: "e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:52:16.692423 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.672577    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-metrics-certs" pod "e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae" (UID: "e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:52:16.768113 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.693674    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-maas-api-tls\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:16.768113 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.693713    4614 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-openshift-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:16.768113 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.693731    4614 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-metrics-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:16.768113 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.693746    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f4gf\" (UniqueName: \"kubernetes.io/projected/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae-kube-api-access-6f4gf\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:16.782961 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.772158    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"]
Nov 02 09:52:16.784248 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.783841    4614 cpu_manager.go:414] "containerMap: removing container" podUID="e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae" containerName="maas-api"
Nov 02 09:52:16.784248 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.783872    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae" containerName="maas-api"
Nov 02 09:52:16.784248 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.784060    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae" containerName="maas-api"
Nov 02 09:52:16.971385 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.951577    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"]
Nov 02 09:52:16.971385 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:16.951724    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.028845 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.004392    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-62e67514-openshift-default-dockercfg-s67mf\""
Nov 02 09:52:17.218984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.218947    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-credential-socket\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.221094 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.221064    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-token\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.221414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.221394    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-podinfo\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.222951 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.222927    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqbtr\" (UniqueName: \"kubernetes.io/projected/9ac06621-1bf7-4939-91bc-c33c0cda61a6-kube-api-access-mqbtr\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.224804 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.224494    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-workload-socket\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.239092 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.234639    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-envoy\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.239092 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.235049    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istiod-ca-cert\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.239092 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.235406    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-data\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.239092 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.235551    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-workload-certs\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.246172 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.246149    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"]
Nov 02 09:52:17.272170 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.263089    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4c342351-8dc476d6c-5jvph"]
Nov 02 09:52:17.316719 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.316673    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae" path="/var/lib/kubelet/pods/e6ecd7a7-acb8-4ea7-91a2-6b2ddaa4f9ae/volumes"
Nov 02 09:52:17.339376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.339244    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-podinfo\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.339376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.339311    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqbtr\" (UniqueName: \"kubernetes.io/projected/9ac06621-1bf7-4939-91bc-c33c0cda61a6-kube-api-access-mqbtr\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.339376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.339377    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-workload-socket\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.339660 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.339401    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-envoy\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.339660 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.339467    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istiod-ca-cert\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.339660 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.339557    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-data\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.339660 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.339586    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-workload-certs\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.339660 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.339628    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-credential-socket\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.339660 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.339644    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-token\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.340824 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.340793    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-workload-certs\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.341127 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.341102    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-data\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.341443 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.341420    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-credential-socket\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.341705 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.341685    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-workload-socket\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.344344 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.344314    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-envoy\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.344510 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.344491    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-podinfo\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.349069 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.349045    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istiod-ca-cert\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.363299 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.363263    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-token\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.371312 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.370019    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqbtr\" (UniqueName: \"kubernetes.io/projected/9ac06621-1bf7-4939-91bc-c33c0cda61a6-kube-api-access-mqbtr\") pod \"e2e-ait-62e67514-openshift-default-6cd4788d5-765tv\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") " pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.460661 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.460534    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n" podUID="847784f5-ed03-4ca8-9450-6546dbee4e8f" containerName="payload-processing" containerID="cri-o://99768c24418c9afb29a4d6c904edac6ff14b36d68eee1e7c4e2f4a623420414c" gracePeriod=25
Nov 02 09:52:17.552377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.542588    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:17.678658 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.678628    4614 generic.go:356] "Generic (PLEG): container finished" podID="847784f5-ed03-4ca8-9450-6546dbee4e8f" containerID="99768c24418c9afb29a4d6c904edac6ff14b36d68eee1e7c4e2f4a623420414c" exitCode=0
Nov 02 09:52:17.678944 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.678673    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n" event={"ID":"847784f5-ed03-4ca8-9450-6546dbee4e8f","Type":"ContainerDied","Data":"99768c24418c9afb29a4d6c904edac6ff14b36d68eee1e7c4e2f4a623420414c"}
Nov 02 09:52:17.885631 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.885586    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"
Nov 02 09:52:17.940518 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.940480    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"]
Nov 02 09:52:17.941581 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:52:17.941553    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac06621_1bf7_4939_91bc_c33c0cda61a6.slice/crio-47f7ac6e23ffdcd0478b3e2a17455d364d78c699dd6452f08ae80df2be822091 WatchSource:0}: Error finding container 47f7ac6e23ffdcd0478b3e2a17455d364d78c699dd6452f08ae80df2be822091: Status 404 returned error can't find the container with id 47f7ac6e23ffdcd0478b3e2a17455d364d78c699dd6452f08ae80df2be822091
Nov 02 09:52:17.945095 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.945062    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:52:17.945199 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.945139    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:52:17.945199 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.945186    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:52:17.953690 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.953657    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/847784f5-ed03-4ca8-9450-6546dbee4e8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/847784f5-ed03-4ca8-9450-6546dbee4e8f-plugins-config-volume\") pod \"847784f5-ed03-4ca8-9450-6546dbee4e8f\" (UID: \"847784f5-ed03-4ca8-9450-6546dbee4e8f\") "
Nov 02 09:52:17.953804 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.953759    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847784f5-ed03-4ca8-9450-6546dbee4e8f-kube-api-access-s48qw\" (UniqueName: \"kubernetes.io/projected/847784f5-ed03-4ca8-9450-6546dbee4e8f-kube-api-access-s48qw\") pod \"847784f5-ed03-4ca8-9450-6546dbee4e8f\" (UID: \"847784f5-ed03-4ca8-9450-6546dbee4e8f\") "
Nov 02 09:52:17.954841 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.954815    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847784f5-ed03-4ca8-9450-6546dbee4e8f-plugins-config-volume" pod "847784f5-ed03-4ca8-9450-6546dbee4e8f" (UID: "847784f5-ed03-4ca8-9450-6546dbee4e8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:52:17.956427 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:17.956399    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847784f5-ed03-4ca8-9450-6546dbee4e8f-kube-api-access-s48qw" pod "847784f5-ed03-4ca8-9450-6546dbee4e8f" (UID: "847784f5-ed03-4ca8-9450-6546dbee4e8f"). InnerVolumeSpecName "kube-api-access-s48qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:18.055880 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:18.055846    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/847784f5-ed03-4ca8-9450-6546dbee4e8f-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:18.055880 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:18.055877    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s48qw\" (UniqueName: \"kubernetes.io/projected/847784f5-ed03-4ca8-9450-6546dbee4e8f-kube-api-access-s48qw\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:18.688649 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:18.688571    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" event={"ID":"9ac06621-1bf7-4939-91bc-c33c0cda61a6","Type":"ContainerStarted","Data":"7530bca637bd6e36c3792fbcdf8d0d08400ab95348796832e85ac0e82217f7e8"}
Nov 02 09:52:18.689166 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:18.689133    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" event={"ID":"9ac06621-1bf7-4939-91bc-c33c0cda61a6","Type":"ContainerStarted","Data":"47f7ac6e23ffdcd0478b3e2a17455d364d78c699dd6452f08ae80df2be822091"}
Nov 02 09:52:18.690107 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:18.690090    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"
Nov 02 09:52:18.690107 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:18.690107    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n" event={"ID":"847784f5-ed03-4ca8-9450-6546dbee4e8f","Type":"ContainerDied","Data":"61691505a0e1fa76e34ebd2813b300bab1d42514ce05fba781a3e4f6b6e87a53"}
Nov 02 09:52:18.690234 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:18.690159    4614 scope.go:122] "RemoveContainer" containerID="99768c24418c9afb29a4d6c904edac6ff14b36d68eee1e7c4e2f4a623420414c"
Nov 02 09:52:18.716737 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:18.716691    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" podStartSLOduration=2.716677041 podStartE2EDuration="2.716677041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:18.713971757 +0000 UTC m=+1019.546629838" watchObservedRunningTime="2026-11-02 09:52:18.716677041 +0000 UTC m=+1019.549335121"
Nov 02 09:52:18.733968 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:18.733940    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"]
Nov 02 09:52:18.736789 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:18.736768    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4c342351-7b78dfb54c-xjx6n"]
Nov 02 09:52:19.025134 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:19.025102    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"]
Nov 02 09:52:19.030583 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:19.030553    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a36b689-w692q"]
Nov 02 09:52:19.316921 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:19.316850    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b7c07ca-855c-4498-bd0f-7dc808ab78cd" path="/var/lib/kubelet/pods/1b7c07ca-855c-4498-bd0f-7dc808ab78cd/volumes"
Nov 02 09:52:19.317254 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:19.317240    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847784f5-ed03-4ca8-9450-6546dbee4e8f" path="/var/lib/kubelet/pods/847784f5-ed03-4ca8-9450-6546dbee4e8f/volumes"
Nov 02 09:52:19.543992 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:19.543945    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:19.549439 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:19.549410    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:19.674512 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:19.674436    4614 scope.go:122] "RemoveContainer" containerID="2a3e1eac805bd9f381441622b690fc27bbda672dd7170ac033a400665d019a77"
Nov 02 09:52:19.685752 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:19.685729    4614 scope.go:122] "RemoveContainer" containerID="470a4f0ca37814600c6156b541fa3e1ac8989344d552f29806cb086ef03045e8"
Nov 02 09:52:19.695033 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:19.694785    4614 scope.go:122] "RemoveContainer" containerID="b43d1d6f3e587f78fcea912e37319647a6f7979e03ffc952be735cca0b044d18"
Nov 02 09:52:19.699639 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:19.699614    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:19.700640 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:19.700624    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:19.703736 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:19.703715    4614 scope.go:122] "RemoveContainer" containerID="1cdff5713a3690c91319ef1450a35eb96c6500fbcc0192cbc09c4421c2596c7c"
Nov 02 09:52:27.389260 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.389227    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"]
Nov 02 09:52:27.389917 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.389900    4614 cpu_manager.go:414] "containerMap: removing container" podUID="847784f5-ed03-4ca8-9450-6546dbee4e8f" containerName="payload-processing"
Nov 02 09:52:27.389969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.389919    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847784f5-ed03-4ca8-9450-6546dbee4e8f" containerName="payload-processing"
Nov 02 09:52:27.390045 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.390032    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="847784f5-ed03-4ca8-9450-6546dbee4e8f" containerName="payload-processing"
Nov 02 09:52:27.400410 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.400377    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:27.404861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.404711    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-62e67514-serving-cert\""
Nov 02 09:52:27.409799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.409775    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"]
Nov 02 09:52:27.499795 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.499761    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba949f46-82f8-4785-8656-922af088f110-maas-api-tls\") pod \"maas-api-e2e-ait-62e67514-77c755f4f-fth8s\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:27.499940 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.499823    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba949f46-82f8-4785-8656-922af088f110-metrics-certs\") pod \"maas-api-e2e-ait-62e67514-77c755f4f-fth8s\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:27.499940 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.499916    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p65s4\" (UniqueName: \"kubernetes.io/projected/ba949f46-82f8-4785-8656-922af088f110-kube-api-access-p65s4\") pod \"maas-api-e2e-ait-62e67514-77c755f4f-fth8s\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:27.500037 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.499974    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba949f46-82f8-4785-8656-922af088f110-openshift-service-ca\") pod \"maas-api-e2e-ait-62e67514-77c755f4f-fth8s\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:27.601380 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.601091    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba949f46-82f8-4785-8656-922af088f110-openshift-service-ca\") pod \"maas-api-e2e-ait-62e67514-77c755f4f-fth8s\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:27.601380 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.601262    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba949f46-82f8-4785-8656-922af088f110-maas-api-tls\") pod \"maas-api-e2e-ait-62e67514-77c755f4f-fth8s\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:27.601380 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.601328    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba949f46-82f8-4785-8656-922af088f110-metrics-certs\") pod \"maas-api-e2e-ait-62e67514-77c755f4f-fth8s\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:27.601707 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.601402    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p65s4\" (UniqueName: \"kubernetes.io/projected/ba949f46-82f8-4785-8656-922af088f110-kube-api-access-p65s4\") pod \"maas-api-e2e-ait-62e67514-77c755f4f-fth8s\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:27.605381 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.602571    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba949f46-82f8-4785-8656-922af088f110-openshift-service-ca\") pod \"maas-api-e2e-ait-62e67514-77c755f4f-fth8s\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:27.614161 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.614124    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba949f46-82f8-4785-8656-922af088f110-metrics-certs\") pod \"maas-api-e2e-ait-62e67514-77c755f4f-fth8s\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:27.644447 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.644338    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p65s4\" (UniqueName: \"kubernetes.io/projected/ba949f46-82f8-4785-8656-922af088f110-kube-api-access-p65s4\") pod \"maas-api-e2e-ait-62e67514-77c755f4f-fth8s\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:27.652076 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.652046    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba949f46-82f8-4785-8656-922af088f110-maas-api-tls\") pod \"maas-api-e2e-ait-62e67514-77c755f4f-fth8s\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:27.721126 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:27.721093    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:28.052713 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:28.052410    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"]
Nov 02 09:52:28.055621 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:52:28.055593    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba949f46_82f8_4785_8656_922af088f110.slice/crio-34903bf2f6b3b96e2c4c5e2dc768c5a0479544eb926d8666a74d2def2f7700c7 WatchSource:0}: Error finding container 34903bf2f6b3b96e2c4c5e2dc768c5a0479544eb926d8666a74d2def2f7700c7: Status 404 returned error can't find the container with id 34903bf2f6b3b96e2c4c5e2dc768c5a0479544eb926d8666a74d2def2f7700c7
Nov 02 09:52:28.057406 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:28.057388    4614 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 09:52:29.025408 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:29.025348    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s" event={"ID":"ba949f46-82f8-4785-8656-922af088f110","Type":"ContainerStarted","Data":"a7bdebe16770dc81fa5ce60b21c84a4e23c866ae15600d564d688daace7e1966"}
Nov 02 09:52:29.025835 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:29.025423    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s" event={"ID":"ba949f46-82f8-4785-8656-922af088f110","Type":"ContainerStarted","Data":"34903bf2f6b3b96e2c4c5e2dc768c5a0479544eb926d8666a74d2def2f7700c7"}
Nov 02 09:52:29.025835 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:29.025581    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:29.047109 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:29.047060    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s" podStartSLOduration=1.493883058 podStartE2EDuration="2.047046998s" totalImagesPullingTime="553.16394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:29.045127586 +0000 UTC m=+1029.877785667" watchObservedRunningTime="2026-11-02 09:52:29.047046998 +0000 UTC m=+1029.879705136"
Nov 02 09:52:35.041562 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:35.041531    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:52:37.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:37.689741    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"]
Nov 02 09:52:37.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:37.690005    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" podUID="3461d99e-f048-49b4-bfa3-6557e546f58b" containerName="istio-proxy" containerID="cri-o://253281eecce78ed19353acd7c12b163c07467640050a48aec07f21ba6439b75e" gracePeriod=30
Nov 02 09:52:39.782565 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:52:39.782525    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:52:41.852959 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:41.852906    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"]
Nov 02 09:52:41.859004 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:41.858981    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"]
Nov 02 09:52:41.859155 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:41.859069    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:41.951293 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:41.951250    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:41.951562 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:41.951326    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz7d6\" (UniqueName: \"kubernetes.io/projected/e67b5ee6-666e-4be6-ad46-b0020f405e7e-kube-api-access-nz7d6\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:41.951562 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:41.951414    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:41.951562 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:41.951444    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:41.951735 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:41.951575    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:41.951735 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:41.951609    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:41.951735 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:41.951657    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:41.951735 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:41.951728    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:41.952056 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:41.951851    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.053529 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.053489    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.053836 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.053816    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz7d6\" (UniqueName: \"kubernetes.io/projected/e67b5ee6-666e-4be6-ad46-b0020f405e7e-kube-api-access-nz7d6\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.053994 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.053980    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.054108 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.054092    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.054265 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.054252    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.054392 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.054378    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.054525 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.054511    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.054662 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.054650    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.054789 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.054760    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.054872 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.054857    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.055066 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.055042    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.055605 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.055579    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.055750 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.055729    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.055909 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.055890    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.057102 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.057080    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.058209 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.058189    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.066134 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.066113    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz7d6\" (UniqueName: \"kubernetes.io/projected/e67b5ee6-666e-4be6-ad46-b0020f405e7e-kube-api-access-nz7d6\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.066247 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.066229    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.177600 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.177511    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:42.619044 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.619020    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"]
Nov 02 09:52:42.620025 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:52:42.619997    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67b5ee6_666e_4be6_ad46_b0020f405e7e.slice/crio-53a5cb462c56e07af146e61375c0e7be97e73d9ff46e5eb88e8b9416378eb635 WatchSource:0}: Error finding container 53a5cb462c56e07af146e61375c0e7be97e73d9ff46e5eb88e8b9416378eb635: Status 404 returned error can't find the container with id 53a5cb462c56e07af146e61375c0e7be97e73d9ff46e5eb88e8b9416378eb635
Nov 02 09:52:42.622630 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.622597    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:52:42.622702 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.622664    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:52:42.622702 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:42.622693    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:52:43.191875 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.191834    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw" event={"ID":"e67b5ee6-666e-4be6-ad46-b0020f405e7e","Type":"ContainerStarted","Data":"f1fd2f816b8746fead0d363ba09930602160f1df9c607b3c82aa3b4a5e540545"}
Nov 02 09:52:43.192323 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.191962    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw" event={"ID":"e67b5ee6-666e-4be6-ad46-b0020f405e7e","Type":"ContainerStarted","Data":"53a5cb462c56e07af146e61375c0e7be97e73d9ff46e5eb88e8b9416378eb635"}
Nov 02 09:52:43.193626 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.193591    4614 generic.go:356] "Generic (PLEG): container finished" podID="3461d99e-f048-49b4-bfa3-6557e546f58b" containerID="253281eecce78ed19353acd7c12b163c07467640050a48aec07f21ba6439b75e" exitCode=0
Nov 02 09:52:43.193769 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.193635    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" event={"ID":"3461d99e-f048-49b4-bfa3-6557e546f58b","Type":"ContainerDied","Data":"253281eecce78ed19353acd7c12b163c07467640050a48aec07f21ba6439b75e"}
Nov 02 09:52:43.217019 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.216967    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw" podStartSLOduration=2.216952702 podStartE2EDuration="2.216952702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:52:43.215005553 +0000 UTC m=+1044.047663646" watchObservedRunningTime="2026-11-02 09:52:43.216952702 +0000 UTC m=+1044.049610783"
Nov 02 09:52:43.603563 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.603509    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:52:43.794743 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.794714    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-workload-certs\") pod \"3461d99e-f048-49b4-bfa3-6557e546f58b\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") "
Nov 02 09:52:43.794969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.794956    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-workload-socket\") pod \"3461d99e-f048-49b4-bfa3-6557e546f58b\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") "
Nov 02 09:52:43.795274 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.795261    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3461d99e-f048-49b4-bfa3-6557e546f58b-kube-api-access-qhhbt\" (UniqueName: \"kubernetes.io/projected/3461d99e-f048-49b4-bfa3-6557e546f58b-kube-api-access-qhhbt\") pod \"3461d99e-f048-49b4-bfa3-6557e546f58b\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") "
Nov 02 09:52:43.795755 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.795735    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-data\") pod \"3461d99e-f048-49b4-bfa3-6557e546f58b\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") "
Nov 02 09:52:43.796233 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.796214    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-token\" (UniqueName: \"kubernetes.io/projected/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-token\") pod \"3461d99e-f048-49b4-bfa3-6557e546f58b\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") "
Nov 02 09:52:43.796737 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.795217    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-workload-socket" pod "3461d99e-f048-49b4-bfa3-6557e546f58b" (UID: "3461d99e-f048-49b4-bfa3-6557e546f58b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:43.796857 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.796104    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-data" pod "3461d99e-f048-49b4-bfa3-6557e546f58b" (UID: "3461d99e-f048-49b4-bfa3-6557e546f58b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:43.796987 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.796450    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-workload-certs" pod "3461d99e-f048-49b4-bfa3-6557e546f58b" (UID: "3461d99e-f048-49b4-bfa3-6557e546f58b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:43.797139 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.797105    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3461d99e-f048-49b4-bfa3-6557e546f58b-istiod-ca-cert" pod "3461d99e-f048-49b4-bfa3-6557e546f58b" (UID: "3461d99e-f048-49b4-bfa3-6557e546f58b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:52:43.797195 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.797109    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3461d99e-f048-49b4-bfa3-6557e546f58b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3461d99e-f048-49b4-bfa3-6557e546f58b-istiod-ca-cert\") pod \"3461d99e-f048-49b4-bfa3-6557e546f58b\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") "
Nov 02 09:52:43.797246 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.797216    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-envoy\") pod \"3461d99e-f048-49b4-bfa3-6557e546f58b\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") "
Nov 02 09:52:43.797294 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.797274    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-credential-socket\") pod \"3461d99e-f048-49b4-bfa3-6557e546f58b\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") "
Nov 02 09:52:43.797391 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.797343    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-podinfo\") pod \"3461d99e-f048-49b4-bfa3-6557e546f58b\" (UID: \"3461d99e-f048-49b4-bfa3-6557e546f58b\") "
Nov 02 09:52:43.797985 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.797962    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:43.798073 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.797987    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:43.798073 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.798000    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3461d99e-f048-49b4-bfa3-6557e546f58b-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:43.798073 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.798013    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:43.799586 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.798915    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-credential-socket" pod "3461d99e-f048-49b4-bfa3-6557e546f58b" (UID: "3461d99e-f048-49b4-bfa3-6557e546f58b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:43.799586 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.799501    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-token" pod "3461d99e-f048-49b4-bfa3-6557e546f58b" (UID: "3461d99e-f048-49b4-bfa3-6557e546f58b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:43.801612 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.799932    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3461d99e-f048-49b4-bfa3-6557e546f58b-kube-api-access-qhhbt" pod "3461d99e-f048-49b4-bfa3-6557e546f58b" (UID: "3461d99e-f048-49b4-bfa3-6557e546f58b"). InnerVolumeSpecName "kube-api-access-qhhbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:52:43.801929 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.801908    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-envoy" pod "3461d99e-f048-49b4-bfa3-6557e546f58b" (UID: "3461d99e-f048-49b4-bfa3-6557e546f58b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:52:43.801929 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.801919    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-podinfo" pod "3461d99e-f048-49b4-bfa3-6557e546f58b" (UID: "3461d99e-f048-49b4-bfa3-6557e546f58b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:52:43.899097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.899011    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:43.899097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.899040    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhhbt\" (UniqueName: \"kubernetes.io/projected/3461d99e-f048-49b4-bfa3-6557e546f58b-kube-api-access-qhhbt\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:43.899097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.899055    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:43.899097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.899068    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:43.899097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:43.899079    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3461d99e-f048-49b4-bfa3-6557e546f58b-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:52:44.188204 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:44.187740    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 09:52:44.188204 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:44.187802    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw" podUID="e67b5ee6-666e-4be6-ad46-b0020f405e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 09:52:44.188204 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:44.188085    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:44.242447 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:44.242406    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x" event={"ID":"3461d99e-f048-49b4-bfa3-6557e546f58b","Type":"ContainerDied","Data":"0144f7d547e811c26c745aebde64b9eae45a657c7755234d7013f4da52137e19"}
Nov 02 09:52:44.242999 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:44.242978    4614 scope.go:122] "RemoveContainer" containerID="253281eecce78ed19353acd7c12b163c07467640050a48aec07f21ba6439b75e"
Nov 02 09:52:44.243230 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:44.242458    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"
Nov 02 09:52:44.395027 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:44.394994    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"]
Nov 02 09:52:44.400652 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:44.400625    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4d57x"]
Nov 02 09:52:45.178571 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:45.178535    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 09:52:45.178760 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:45.178592    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw" podUID="e67b5ee6-666e-4be6-ad46-b0020f405e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 09:52:45.324161 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:45.324122    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3461d99e-f048-49b4-bfa3-6557e546f58b" path="/var/lib/kubelet/pods/3461d99e-f048-49b4-bfa3-6557e546f58b/volumes"
Nov 02 09:52:46.178152 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:46.178116    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 09:52:46.178346 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:46.178191    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw" podUID="e67b5ee6-666e-4be6-ad46-b0020f405e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 09:52:47.178565 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:47.178531    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 09:52:47.178983 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:47.178609    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw" podUID="e67b5ee6-666e-4be6-ad46-b0020f405e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 09:52:48.177827 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:48.177793    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 09:52:48.178029 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:48.177842    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw" podUID="e67b5ee6-666e-4be6-ad46-b0020f405e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 09:52:49.178758 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:49.178718    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 02 09:52:49.179169 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:49.178785    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw" podUID="e67b5ee6-666e-4be6-ad46-b0020f405e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 02 09:52:49.732244 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:49.730158    4614 patch_prober.go:29] interesting pod/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": read tcp 10.132.0.2:48764->10.132.0.96:15021: read: connection reset by peer" start-of-body=
Nov 02 09:52:49.732244 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:49.730203    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": read tcp 10.132.0.2:48764->10.132.0.96:15021: read: connection reset by peer"
Nov 02 09:52:50.182520 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:50.182446    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:50.182874 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:50.182739    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:50.183392 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:50.183346    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:52:50.321505 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:50.321476    4614 generic.go:356] "Generic (PLEG): container finished" podID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerID="7530bca637bd6e36c3792fbcdf8d0d08400ab95348796832e85ac0e82217f7e8" exitCode=0
Nov 02 09:52:50.321704 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:50.321547    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" event={"ID":"9ac06621-1bf7-4939-91bc-c33c0cda61a6","Type":"ContainerDied","Data":"7530bca637bd6e36c3792fbcdf8d0d08400ab95348796832e85ac0e82217f7e8"}
Nov 02 09:52:50.322031 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:50.322015    4614 scope.go:122] "RemoveContainer" containerID="7530bca637bd6e36c3792fbcdf8d0d08400ab95348796832e85ac0e82217f7e8"
Nov 02 09:52:50.323242 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:50.323216    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:52:50.323295 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:50.323284    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:52:50.323388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:50.323319    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:52:51.349376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:51.347784    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" event={"ID":"9ac06621-1bf7-4939-91bc-c33c0cda61a6","Type":"ContainerStarted","Data":"aaa1e9f9e187ba4140fa721552dc92fa326db8a0d126685163c4937451fdc9d9"}
Nov 02 09:52:51.547776 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:51.547742    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:51.550507 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:51.549039    4614 patch_prober.go:29] interesting pod/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 09:52:51.550739 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:51.550714    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 09:52:52.557377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:52.550835    4614 patch_prober.go:29] interesting pod/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 09:52:52.557377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:52.550930    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 09:52:53.339534 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:53.339489    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:52:53.547378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:53.544989    4614 patch_prober.go:29] interesting pod/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 09:52:53.547378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:53.545052    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 09:52:53.713559 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:53.713448    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:52:53.724494 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:53.724458    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:52:54.546375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:54.544327    4614 patch_prober.go:29] interesting pod/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 09:52:54.546375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:54.544418    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 09:52:55.552624 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:55.551453    4614 patch_prober.go:29] interesting pod/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 09:52:55.552624 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:55.551529    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 09:52:56.552242 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:56.551769    4614 patch_prober.go:29] interesting pod/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 09:52:56.552242 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:56.551828    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 09:52:57.568162 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:57.568121    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:57.568743 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:57.568720    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:57.569994 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:57.569968    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:52:58.025824 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:58.025791    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"]
Nov 02 09:52:58.028533 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:58.028509    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164501f8-sggxm"]
Nov 02 09:52:59.315949 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:59.315914    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c18ead-e49f-46a8-8dc8-7a652bd8c774" path="/var/lib/kubelet/pods/92c18ead-e49f-46a8-8dc8-7a652bd8c774/volumes"
Nov 02 09:52:59.682315 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:59.682229    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"]
Nov 02 09:52:59.682955 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:52:59.682919    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s" podUID="ba949f46-82f8-4785-8656-922af088f110" containerName="maas-api" containerID="cri-o://a7bdebe16770dc81fa5ce60b21c84a4e23c866ae15600d564d688daace7e1966" gracePeriod=30
Nov 02 09:53:00.012382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.010142    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:53:00.202417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.201968    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba949f46-82f8-4785-8656-922af088f110-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba949f46-82f8-4785-8656-922af088f110-metrics-certs\") pod \"ba949f46-82f8-4785-8656-922af088f110\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") "
Nov 02 09:53:00.202417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.202066    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba949f46-82f8-4785-8656-922af088f110-kube-api-access-p65s4\" (UniqueName: \"kubernetes.io/projected/ba949f46-82f8-4785-8656-922af088f110-kube-api-access-p65s4\") pod \"ba949f46-82f8-4785-8656-922af088f110\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") "
Nov 02 09:53:00.202417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.202159    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba949f46-82f8-4785-8656-922af088f110-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba949f46-82f8-4785-8656-922af088f110-maas-api-tls\") pod \"ba949f46-82f8-4785-8656-922af088f110\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") "
Nov 02 09:53:00.202417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.202242    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba949f46-82f8-4785-8656-922af088f110-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba949f46-82f8-4785-8656-922af088f110-openshift-service-ca\") pod \"ba949f46-82f8-4785-8656-922af088f110\" (UID: \"ba949f46-82f8-4785-8656-922af088f110\") "
Nov 02 09:53:00.204821 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.204331    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba949f46-82f8-4785-8656-922af088f110-openshift-service-ca" pod "ba949f46-82f8-4785-8656-922af088f110" (UID: "ba949f46-82f8-4785-8656-922af088f110"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:00.218658 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.211546    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba949f46-82f8-4785-8656-922af088f110-kube-api-access-p65s4" pod "ba949f46-82f8-4785-8656-922af088f110" (UID: "ba949f46-82f8-4785-8656-922af088f110"). InnerVolumeSpecName "kube-api-access-p65s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:00.218658 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.212396    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba949f46-82f8-4785-8656-922af088f110-maas-api-tls" pod "ba949f46-82f8-4785-8656-922af088f110" (UID: "ba949f46-82f8-4785-8656-922af088f110"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:53:00.292376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.276319    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba949f46-82f8-4785-8656-922af088f110-metrics-certs" pod "ba949f46-82f8-4785-8656-922af088f110" (UID: "ba949f46-82f8-4785-8656-922af088f110"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:53:00.334893 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.334859    4614 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba949f46-82f8-4785-8656-922af088f110-openshift-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:00.334893 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.334896    4614 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba949f46-82f8-4785-8656-922af088f110-metrics-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:00.335461 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.334913    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p65s4\" (UniqueName: \"kubernetes.io/projected/ba949f46-82f8-4785-8656-922af088f110-kube-api-access-p65s4\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:00.335461 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.334934    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba949f46-82f8-4785-8656-922af088f110-maas-api-tls\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:00.490378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.487105    4614 generic.go:356] "Generic (PLEG): container finished" podID="ba949f46-82f8-4785-8656-922af088f110" containerID="a7bdebe16770dc81fa5ce60b21c84a4e23c866ae15600d564d688daace7e1966" exitCode=0
Nov 02 09:53:00.490929 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.490695    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"
Nov 02 09:53:00.520425 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.490719    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s" event={"ID":"ba949f46-82f8-4785-8656-922af088f110","Type":"ContainerDied","Data":"a7bdebe16770dc81fa5ce60b21c84a4e23c866ae15600d564d688daace7e1966"}
Nov 02 09:53:00.520425 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.509788    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s" event={"ID":"ba949f46-82f8-4785-8656-922af088f110","Type":"ContainerDied","Data":"34903bf2f6b3b96e2c4c5e2dc768c5a0479544eb926d8666a74d2def2f7700c7"}
Nov 02 09:53:00.520425 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.509830    4614 scope.go:122] "RemoveContainer" containerID="a7bdebe16770dc81fa5ce60b21c84a4e23c866ae15600d564d688daace7e1966"
Nov 02 09:53:00.576196 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.576081    4614 scope.go:122] "RemoveContainer" containerID="a7bdebe16770dc81fa5ce60b21c84a4e23c866ae15600d564d688daace7e1966"
Nov 02 09:53:00.589376 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:53:00.589333    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7bdebe16770dc81fa5ce60b21c84a4e23c866ae15600d564d688daace7e1966\": container with ID starting with a7bdebe16770dc81fa5ce60b21c84a4e23c866ae15600d564d688daace7e1966 not found: ID does not exist" containerID="a7bdebe16770dc81fa5ce60b21c84a4e23c866ae15600d564d688daace7e1966"
Nov 02 09:53:00.589614 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.589576    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bdebe16770dc81fa5ce60b21c84a4e23c866ae15600d564d688daace7e1966"} err="failed to get container status \"a7bdebe16770dc81fa5ce60b21c84a4e23c866ae15600d564d688daace7e1966\": rpc error: code = NotFound desc = could not find container \"a7bdebe16770dc81fa5ce60b21c84a4e23c866ae15600d564d688daace7e1966\": container with ID starting with a7bdebe16770dc81fa5ce60b21c84a4e23c866ae15600d564d688daace7e1966 not found: ID does not exist"
Nov 02 09:53:00.591493 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.591460    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"]
Nov 02 09:53:00.597079 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:00.597053    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-62e67514-77c755f4f-fth8s"]
Nov 02 09:53:01.349923 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:01.349888    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba949f46-82f8-4785-8656-922af088f110" path="/var/lib/kubelet/pods/ba949f46-82f8-4785-8656-922af088f110/volumes"
Nov 02 09:53:01.607860 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:01.606719    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"]
Nov 02 09:53:01.607860 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:01.606954    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerName="istio-proxy" containerID="cri-o://aaa1e9f9e187ba4140fa721552dc92fa326db8a0d126685163c4937451fdc9d9" gracePeriod=30
Nov 02 09:53:02.404511 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:02.404468    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:53:02.405128 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:02.405090    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:53:04.045180 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.045141    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"]
Nov 02 09:53:04.045678 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.045664    4614 cpu_manager.go:414] "containerMap: removing container" podUID="3461d99e-f048-49b4-bfa3-6557e546f58b" containerName="istio-proxy"
Nov 02 09:53:04.045725 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.045679    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3461d99e-f048-49b4-bfa3-6557e546f58b" containerName="istio-proxy"
Nov 02 09:53:04.045725 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.045696    4614 cpu_manager.go:414] "containerMap: removing container" podUID="ba949f46-82f8-4785-8656-922af088f110" containerName="maas-api"
Nov 02 09:53:04.045725 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.045702    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba949f46-82f8-4785-8656-922af088f110" containerName="maas-api"
Nov 02 09:53:04.045821 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.045782    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="3461d99e-f048-49b4-bfa3-6557e546f58b" containerName="istio-proxy"
Nov 02 09:53:04.045821 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.045792    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba949f46-82f8-4785-8656-922af088f110" containerName="maas-api"
Nov 02 09:53:04.050590 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.050571    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.068448 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.068429    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9cbd331c-openshift-default-dockercfg-t4z5f\""
Nov 02 09:53:04.079212 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.079187    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"]
Nov 02 09:53:04.206609 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.206545    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-workload-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.208443 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.208396    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-istio-data\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.208601 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.208460    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b48362a-5918-4899-abac-14f674804556-istio-token\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.208672 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.208635    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-istio-envoy\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.208730 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.208677    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b48362a-5918-4899-abac-14f674804556-istiod-ca-cert\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.208806 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.208786    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b48362a-5918-4899-abac-14f674804556-istio-podinfo\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.208907 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.208888    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-workload-certs\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.208967 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.208934    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-credential-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.209118 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.209097    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbb8p\" (UniqueName: \"kubernetes.io/projected/7b48362a-5918-4899-abac-14f674804556-kube-api-access-nbb8p\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.310164 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.310086    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-istio-envoy\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.310164 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.310125    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b48362a-5918-4899-abac-14f674804556-istiod-ca-cert\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.310339 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.310167    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b48362a-5918-4899-abac-14f674804556-istio-podinfo\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.310339 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.310235    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-workload-certs\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.310339 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.310272    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-credential-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.310510 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.310411    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbb8p\" (UniqueName: \"kubernetes.io/projected/7b48362a-5918-4899-abac-14f674804556-kube-api-access-nbb8p\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.310510 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.310469    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-workload-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.310510 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.310496    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-istio-data\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.310655 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.310525    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b48362a-5918-4899-abac-14f674804556-istio-token\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.310707 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.310643    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-workload-certs\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.310757 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.310704    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-credential-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.310950 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.310918    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-workload-socket\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.311095 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.310997    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-istio-data\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.311166 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.311110    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b48362a-5918-4899-abac-14f674804556-istiod-ca-cert\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.312641 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.312618    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-istio-envoy\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.312735 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.312674    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b48362a-5918-4899-abac-14f674804556-istio-podinfo\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.323687 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.323664    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b48362a-5918-4899-abac-14f674804556-istio-token\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.323795 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.323722    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbb8p\" (UniqueName: \"kubernetes.io/projected/7b48362a-5918-4899-abac-14f674804556-kube-api-access-nbb8p\") pod \"e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") " pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.364317 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.364288    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:04.659672 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.659641    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"]
Nov 02 09:53:04.661029 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:53:04.661004    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b48362a_5918_4899_abac_14f674804556.slice/crio-5816cc045c68ed0e387767065118bd07bd5c9eba1698ea69a5a23617be473b5b WatchSource:0}: Error finding container 5816cc045c68ed0e387767065118bd07bd5c9eba1698ea69a5a23617be473b5b: Status 404 returned error can't find the container with id 5816cc045c68ed0e387767065118bd07bd5c9eba1698ea69a5a23617be473b5b
Nov 02 09:53:04.663636 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.663607    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:53:04.663727 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.663671    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:53:04.663727 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:04.663699    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:53:05.245930 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:05.244285    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7363-2b9420"]
Nov 02 09:53:05.269187 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:05.269156    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7363-2b9420"]
Nov 02 09:53:05.272291 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:05.269446    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7363-2b9420"
Nov 02 09:53:05.277739 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:05.277695    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x8bqk\""
Nov 02 09:53:05.453310 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:05.453258    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwqn6\" (UniqueName: \"kubernetes.io/projected/32a247c5-893a-4161-aee6-eb413d412f3f-kube-api-access-cwqn6\") pod \"test-curl-7363-2b9420\" (UID: \"32a247c5-893a-4161-aee6-eb413d412f3f\") " pod="opendatahub/test-curl-7363-2b9420"
Nov 02 09:53:05.556289 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:05.556182    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwqn6\" (UniqueName: \"kubernetes.io/projected/32a247c5-893a-4161-aee6-eb413d412f3f-kube-api-access-cwqn6\") pod \"test-curl-7363-2b9420\" (UID: \"32a247c5-893a-4161-aee6-eb413d412f3f\") " pod="opendatahub/test-curl-7363-2b9420"
Nov 02 09:53:05.601526 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:05.595973    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwqn6\" (UniqueName: \"kubernetes.io/projected/32a247c5-893a-4161-aee6-eb413d412f3f-kube-api-access-cwqn6\") pod \"test-curl-7363-2b9420\" (UID: \"32a247c5-893a-4161-aee6-eb413d412f3f\") " pod="opendatahub/test-curl-7363-2b9420"
Nov 02 09:53:05.601526 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:05.596194    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7363-2b9420"
Nov 02 09:53:05.679071 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:05.679022    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j" event={"ID":"7b48362a-5918-4899-abac-14f674804556","Type":"ContainerStarted","Data":"28e175dbd047d908170c47865844ddc2cc9aa190d5c55b0e61f3d44b6d6da72c"}
Nov 02 09:53:05.679335 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:05.679301    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j" event={"ID":"7b48362a-5918-4899-abac-14f674804556","Type":"ContainerStarted","Data":"5816cc045c68ed0e387767065118bd07bd5c9eba1698ea69a5a23617be473b5b"}
Nov 02 09:53:05.794629 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:05.794557    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j" podStartSLOduration=1.7945351459999999 podStartE2EDuration="1.794535146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:05.785073 +0000 UTC m=+1066.617731085" watchObservedRunningTime="2026-11-02 09:53:05.794535146 +0000 UTC m=+1066.627193228"
Nov 02 09:53:06.069046 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:53:06.069009    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32a247c5_893a_4161_aee6_eb413d412f3f.slice/crio-8ac0790dfa6dcd7026b68cf138c9974d7f8969eb90bb2546fd7d7353ad42d2de WatchSource:0}: Error finding container 8ac0790dfa6dcd7026b68cf138c9974d7f8969eb90bb2546fd7d7353ad42d2de: Status 404 returned error can't find the container with id 8ac0790dfa6dcd7026b68cf138c9974d7f8969eb90bb2546fd7d7353ad42d2de
Nov 02 09:53:06.069215 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:06.069117    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7363-2b9420"]
Nov 02 09:53:06.364517 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:06.364481    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:06.366238 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:06.366211    4614 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 02 09:53:06.366439 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:06.366263    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j" podUID="7b48362a-5918-4899-abac-14f674804556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 02 09:53:06.686232 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:06.686148    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7363-2b9420" event={"ID":"32a247c5-893a-4161-aee6-eb413d412f3f","Type":"ContainerStarted","Data":"8ac0790dfa6dcd7026b68cf138c9974d7f8969eb90bb2546fd7d7353ad42d2de"}
Nov 02 09:53:07.368812 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.368763    4614 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 02 09:53:07.369451 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.369416    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j" podUID="7b48362a-5918-4899-abac-14f674804556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 02 09:53:07.401269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.401245    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:53:07.423672 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.423647    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-token\" (UniqueName: \"kubernetes.io/projected/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-token\") pod \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") "
Nov 02 09:53:07.424269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.423710    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-envoy\") pod \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") "
Nov 02 09:53:07.424269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.423830    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-data\") pod \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") "
Nov 02 09:53:07.424269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.423861    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-workload-certs\") pod \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") "
Nov 02 09:53:07.424269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.423977    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-workload-socket\") pod \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") "
Nov 02 09:53:07.424269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.424066    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-credential-socket\") pod \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") "
Nov 02 09:53:07.424269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.424165    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istiod-ca-cert\") pod \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") "
Nov 02 09:53:07.424269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.424200    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-data" pod "9ac06621-1bf7-4939-91bc-c33c0cda61a6" (UID: "9ac06621-1bf7-4939-91bc-c33c0cda61a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:53:07.424269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.424216    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac06621-1bf7-4939-91bc-c33c0cda61a6-kube-api-access-mqbtr\" (UniqueName: \"kubernetes.io/projected/9ac06621-1bf7-4939-91bc-c33c0cda61a6-kube-api-access-mqbtr\") pod \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") "
Nov 02 09:53:07.424269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.424262    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-podinfo\") pod \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\" (UID: \"9ac06621-1bf7-4939-91bc-c33c0cda61a6\") "
Nov 02 09:53:07.425491 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.424437    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-workload-socket" pod "9ac06621-1bf7-4939-91bc-c33c0cda61a6" (UID: "9ac06621-1bf7-4939-91bc-c33c0cda61a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:53:07.425491 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.424772    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-workload-certs" pod "9ac06621-1bf7-4939-91bc-c33c0cda61a6" (UID: "9ac06621-1bf7-4939-91bc-c33c0cda61a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:53:07.425491 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.425284    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istiod-ca-cert" pod "9ac06621-1bf7-4939-91bc-c33c0cda61a6" (UID: "9ac06621-1bf7-4939-91bc-c33c0cda61a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:07.454642 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.454584    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-credential-socket" pod "9ac06621-1bf7-4939-91bc-c33c0cda61a6" (UID: "9ac06621-1bf7-4939-91bc-c33c0cda61a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:53:07.455817 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.455778    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-envoy" pod "9ac06621-1bf7-4939-91bc-c33c0cda61a6" (UID: "9ac06621-1bf7-4939-91bc-c33c0cda61a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:53:07.456084 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.456054    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac06621-1bf7-4939-91bc-c33c0cda61a6-kube-api-access-mqbtr" pod "9ac06621-1bf7-4939-91bc-c33c0cda61a6" (UID: "9ac06621-1bf7-4939-91bc-c33c0cda61a6"). InnerVolumeSpecName "kube-api-access-mqbtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:07.456206 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.456171    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:07.456275 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.456204    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:07.456275 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.456222    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:07.456275 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.456237    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:07.460844 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.460788    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-token" pod "9ac06621-1bf7-4939-91bc-c33c0cda61a6" (UID: "9ac06621-1bf7-4939-91bc-c33c0cda61a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:07.465482 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.465452    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-podinfo" pod "9ac06621-1bf7-4939-91bc-c33c0cda61a6" (UID: "9ac06621-1bf7-4939-91bc-c33c0cda61a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:53:07.557319 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.557232    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqbtr\" (UniqueName: \"kubernetes.io/projected/9ac06621-1bf7-4939-91bc-c33c0cda61a6-kube-api-access-mqbtr\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:07.557319 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.557273    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:07.557319 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.557289    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:07.557319 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.557304    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:07.557319 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.557318    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac06621-1bf7-4939-91bc-c33c0cda61a6-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:07.693908 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.693872    4614 generic.go:356] "Generic (PLEG): container finished" podID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerID="aaa1e9f9e187ba4140fa721552dc92fa326db8a0d126685163c4937451fdc9d9" exitCode=0
Nov 02 09:53:07.694106 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.693923    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"
Nov 02 09:53:07.694106 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.693947    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" event={"ID":"9ac06621-1bf7-4939-91bc-c33c0cda61a6","Type":"ContainerDied","Data":"aaa1e9f9e187ba4140fa721552dc92fa326db8a0d126685163c4937451fdc9d9"}
Nov 02 09:53:07.694106 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.694010    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv" event={"ID":"9ac06621-1bf7-4939-91bc-c33c0cda61a6","Type":"ContainerDied","Data":"47f7ac6e23ffdcd0478b3e2a17455d364d78c699dd6452f08ae80df2be822091"}
Nov 02 09:53:07.694106 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.694041    4614 scope.go:122] "RemoveContainer" containerID="aaa1e9f9e187ba4140fa721552dc92fa326db8a0d126685163c4937451fdc9d9"
Nov 02 09:53:07.705557 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.705538    4614 scope.go:122] "RemoveContainer" containerID="7530bca637bd6e36c3792fbcdf8d0d08400ab95348796832e85ac0e82217f7e8"
Nov 02 09:53:07.714511 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.714492    4614 scope.go:122] "RemoveContainer" containerID="aaa1e9f9e187ba4140fa721552dc92fa326db8a0d126685163c4937451fdc9d9"
Nov 02 09:53:07.714781 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:53:07.714762    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa1e9f9e187ba4140fa721552dc92fa326db8a0d126685163c4937451fdc9d9\": container with ID starting with aaa1e9f9e187ba4140fa721552dc92fa326db8a0d126685163c4937451fdc9d9 not found: ID does not exist" containerID="aaa1e9f9e187ba4140fa721552dc92fa326db8a0d126685163c4937451fdc9d9"
Nov 02 09:53:07.714858 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.714794    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa1e9f9e187ba4140fa721552dc92fa326db8a0d126685163c4937451fdc9d9"} err="failed to get container status \"aaa1e9f9e187ba4140fa721552dc92fa326db8a0d126685163c4937451fdc9d9\": rpc error: code = NotFound desc = could not find container \"aaa1e9f9e187ba4140fa721552dc92fa326db8a0d126685163c4937451fdc9d9\": container with ID starting with aaa1e9f9e187ba4140fa721552dc92fa326db8a0d126685163c4937451fdc9d9 not found: ID does not exist"
Nov 02 09:53:07.714858 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.714818    4614 scope.go:122] "RemoveContainer" containerID="7530bca637bd6e36c3792fbcdf8d0d08400ab95348796832e85ac0e82217f7e8"
Nov 02 09:53:07.715042 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:53:07.715024    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7530bca637bd6e36c3792fbcdf8d0d08400ab95348796832e85ac0e82217f7e8\": container with ID starting with 7530bca637bd6e36c3792fbcdf8d0d08400ab95348796832e85ac0e82217f7e8 not found: ID does not exist" containerID="7530bca637bd6e36c3792fbcdf8d0d08400ab95348796832e85ac0e82217f7e8"
Nov 02 09:53:07.715095 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.715050    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7530bca637bd6e36c3792fbcdf8d0d08400ab95348796832e85ac0e82217f7e8"} err="failed to get container status \"7530bca637bd6e36c3792fbcdf8d0d08400ab95348796832e85ac0e82217f7e8\": rpc error: code = NotFound desc = could not find container \"7530bca637bd6e36c3792fbcdf8d0d08400ab95348796832e85ac0e82217f7e8\": container with ID starting with 7530bca637bd6e36c3792fbcdf8d0d08400ab95348796832e85ac0e82217f7e8 not found: ID does not exist"
Nov 02 09:53:07.719482 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.719457    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"]
Nov 02 09:53:07.724377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:07.724340    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-62e67514-openshift-default-6cd4788d5-765tv"]
Nov 02 09:53:08.364757 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:08.364716    4614 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 02 09:53:08.364945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:08.364780    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j" podUID="7b48362a-5918-4899-abac-14f674804556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 02 09:53:09.318106 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:09.318069    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" path="/var/lib/kubelet/pods/9ac06621-1bf7-4939-91bc-c33c0cda61a6/volumes"
Nov 02 09:53:09.366071 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:09.366038    4614 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 02 09:53:09.366242 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:09.366090    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j" podUID="7b48362a-5918-4899-abac-14f674804556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 02 09:53:09.662711 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:53:09.662624    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:53:10.365159 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:10.365121    4614 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 02 09:53:10.365636 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:10.365178    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j" podUID="7b48362a-5918-4899-abac-14f674804556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 02 09:53:11.365886 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:11.365847    4614 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 02 09:53:11.366423 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:11.365922    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j" podUID="7b48362a-5918-4899-abac-14f674804556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 02 09:53:12.369549 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:12.369519    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:12.370038 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:12.369930    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:12.370815 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:12.370783    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:53:15.696006 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:15.695492    4614 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:53:15.711589 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:15.710432    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:53:18.838763 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:18.838722    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7363-2b9420" event={"ID":"32a247c5-893a-4161-aee6-eb413d412f3f","Type":"ContainerStarted","Data":"5bfb5dba895e49fa24b8fc6fe8eefa67d4e8df770cb439a7251eb44c11179fc0"}
Nov 02 09:53:18.855212 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:18.855160    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7363-2b9420" podStartSLOduration=2.114798694 podStartE2EDuration="13.855148252s" totalImagesPullingTime="11.740349558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:18.854158674 +0000 UTC m=+1079.686816754" watchObservedRunningTime="2026-11-02 09:53:18.855148252 +0000 UTC m=+1079.687806331"
Nov 02 09:53:19.559282 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:19.559249    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7363-2b9420"]
Nov 02 09:53:19.566133 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:19.566104    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7363-2b9420"]
Nov 02 09:53:19.774481 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:19.774446    4614 scope.go:122] "RemoveContainer" containerID="29d9f91790c567f3b7eb6bd48012ef940ee4d8136e410c0ead11e55820bbf8dc"
Nov 02 09:53:19.797576 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:19.797551    4614 scope.go:122] "RemoveContainer" containerID="bb7a9ec02c01b1c597d6ff2f9899f21ace320eef982a9d2632a2bf75f0efab89"
Nov 02 09:53:19.806424 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:19.806405    4614 scope.go:122] "RemoveContainer" containerID="2ef7d7570cfba1592e85f1fde35f438e5ab6823f7bf1f94a427456c0ccf8f1bf"
Nov 02 09:53:19.814946 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:19.814927    4614 scope.go:122] "RemoveContainer" containerID="2915783aead53101faa0ba561b33b6d33445dd493921291a99c227349b3cff3e"
Nov 02 09:53:19.824060 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:19.824044    4614 scope.go:122] "RemoveContainer" containerID="bbf86b81537eb490736edd2d96089282f9540ad59ffd02e3885e11c3ea4b0d57"
Nov 02 09:53:20.858744 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:20.858704    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7363-2b9420" podUID="32a247c5-893a-4161-aee6-eb413d412f3f" containerName="test-curl-7363-2b9420" containerID="cri-o://5bfb5dba895e49fa24b8fc6fe8eefa67d4e8df770cb439a7251eb44c11179fc0" gracePeriod=2
Nov 02 09:53:20.861189 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:53:20.861160    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7363-2b9420\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="32a247c5-893a-4161-aee6-eb413d412f3f" pod="opendatahub/test-curl-7363-2b9420"
Nov 02 09:53:23.075450 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.075428    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7363-2b9420"
Nov 02 09:53:23.187609 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.187525    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a247c5-893a-4161-aee6-eb413d412f3f-kube-api-access-cwqn6\" (UniqueName: \"kubernetes.io/projected/32a247c5-893a-4161-aee6-eb413d412f3f-kube-api-access-cwqn6\") pod \"32a247c5-893a-4161-aee6-eb413d412f3f\" (UID: \"32a247c5-893a-4161-aee6-eb413d412f3f\") "
Nov 02 09:53:23.190060 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.190030    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a247c5-893a-4161-aee6-eb413d412f3f-kube-api-access-cwqn6" pod "32a247c5-893a-4161-aee6-eb413d412f3f" (UID: "32a247c5-893a-4161-aee6-eb413d412f3f"). InnerVolumeSpecName "kube-api-access-cwqn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:23.289316 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.289279    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwqn6\" (UniqueName: \"kubernetes.io/projected/32a247c5-893a-4161-aee6-eb413d412f3f-kube-api-access-cwqn6\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:23.316362 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.316326    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a247c5-893a-4161-aee6-eb413d412f3f" path="/var/lib/kubelet/pods/32a247c5-893a-4161-aee6-eb413d412f3f/volumes"
Nov 02 09:53:23.345097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.345061    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"]
Nov 02 09:53:23.345631 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.345612    4614 cpu_manager.go:414] "containerMap: removing container" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerName="istio-proxy"
Nov 02 09:53:23.345746 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.345632    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerName="istio-proxy"
Nov 02 09:53:23.345746 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.345656    4614 cpu_manager.go:414] "containerMap: removing container" podUID="32a247c5-893a-4161-aee6-eb413d412f3f" containerName="test-curl-7363-2b9420"
Nov 02 09:53:23.345746 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.345664    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a247c5-893a-4161-aee6-eb413d412f3f" containerName="test-curl-7363-2b9420"
Nov 02 09:53:23.345746 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.345694    4614 cpu_manager.go:414] "containerMap: removing container" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerName="istio-proxy"
Nov 02 09:53:23.345746 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.345702    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerName="istio-proxy"
Nov 02 09:53:23.345997 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.345871    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerName="istio-proxy"
Nov 02 09:53:23.345997 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.345887    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a247c5-893a-4161-aee6-eb413d412f3f" containerName="test-curl-7363-2b9420"
Nov 02 09:53:23.350174 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.350154    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"
Nov 02 09:53:23.356627 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.356608    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:53:23.356884 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.356867    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:53:23.361045 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.361020    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"]
Nov 02 09:53:23.390412 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.390384    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4fpm\" (UniqueName: \"kubernetes.io/projected/03857226-641c-4716-948b-fe13d3b787c3-kube-api-access-t4fpm\") pod \"maas-api-revoke-sub-f2604498-t9jqt\" (UID: \"03857226-641c-4716-948b-fe13d3b787c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"
Nov 02 09:53:23.390556 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.390468    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03857226-641c-4716-948b-fe13d3b787c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2604498-t9jqt\" (UID: \"03857226-641c-4716-948b-fe13d3b787c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"
Nov 02 09:53:23.390556 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.390519    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03857226-641c-4716-948b-fe13d3b787c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2604498-t9jqt\" (UID: \"03857226-641c-4716-948b-fe13d3b787c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"
Nov 02 09:53:23.491549 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.491520    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4fpm\" (UniqueName: \"kubernetes.io/projected/03857226-641c-4716-948b-fe13d3b787c3-kube-api-access-t4fpm\") pod \"maas-api-revoke-sub-f2604498-t9jqt\" (UID: \"03857226-641c-4716-948b-fe13d3b787c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"
Nov 02 09:53:23.491736 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.491575    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03857226-641c-4716-948b-fe13d3b787c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2604498-t9jqt\" (UID: \"03857226-641c-4716-948b-fe13d3b787c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"
Nov 02 09:53:23.491736 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.491606    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03857226-641c-4716-948b-fe13d3b787c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2604498-t9jqt\" (UID: \"03857226-641c-4716-948b-fe13d3b787c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"
Nov 02 09:53:23.492296 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.492273    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03857226-641c-4716-948b-fe13d3b787c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2604498-t9jqt\" (UID: \"03857226-641c-4716-948b-fe13d3b787c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"
Nov 02 09:53:23.494089 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.494064    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03857226-641c-4716-948b-fe13d3b787c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2604498-t9jqt\" (UID: \"03857226-641c-4716-948b-fe13d3b787c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"
Nov 02 09:53:23.503780 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.503760    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4fpm\" (UniqueName: \"kubernetes.io/projected/03857226-641c-4716-948b-fe13d3b787c3-kube-api-access-t4fpm\") pod \"maas-api-revoke-sub-f2604498-t9jqt\" (UID: \"03857226-641c-4716-948b-fe13d3b787c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"
Nov 02 09:53:23.666290 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.666252    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"
Nov 02 09:53:23.878530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.878447    4614 generic.go:356] "Generic (PLEG): container finished" podID="32a247c5-893a-4161-aee6-eb413d412f3f" containerID="5bfb5dba895e49fa24b8fc6fe8eefa67d4e8df770cb439a7251eb44c11179fc0" exitCode=137
Nov 02 09:53:23.878530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.878501    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7363-2b9420"
Nov 02 09:53:23.878782 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.878547    4614 scope.go:122] "RemoveContainer" containerID="5bfb5dba895e49fa24b8fc6fe8eefa67d4e8df770cb439a7251eb44c11179fc0"
Nov 02 09:53:23.887980 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.887962    4614 scope.go:122] "RemoveContainer" containerID="5bfb5dba895e49fa24b8fc6fe8eefa67d4e8df770cb439a7251eb44c11179fc0"
Nov 02 09:53:23.888300 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:53:23.888260    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bfb5dba895e49fa24b8fc6fe8eefa67d4e8df770cb439a7251eb44c11179fc0\": container with ID starting with 5bfb5dba895e49fa24b8fc6fe8eefa67d4e8df770cb439a7251eb44c11179fc0 not found: ID does not exist" containerID="5bfb5dba895e49fa24b8fc6fe8eefa67d4e8df770cb439a7251eb44c11179fc0"
Nov 02 09:53:23.888300 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.888283    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bfb5dba895e49fa24b8fc6fe8eefa67d4e8df770cb439a7251eb44c11179fc0"} err="failed to get container status \"5bfb5dba895e49fa24b8fc6fe8eefa67d4e8df770cb439a7251eb44c11179fc0\": rpc error: code = NotFound desc = could not find container \"5bfb5dba895e49fa24b8fc6fe8eefa67d4e8df770cb439a7251eb44c11179fc0\": container with ID starting with 5bfb5dba895e49fa24b8fc6fe8eefa67d4e8df770cb439a7251eb44c11179fc0 not found: ID does not exist"
Nov 02 09:53:23.949992 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:23.949961    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"]
Nov 02 09:53:23.951797 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:53:23.951769    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03857226_641c_4716_948b_fe13d3b787c3.slice/crio-d9b3f41857ed89e4f00c922ba85ee7a7fa45e31e0f10133705df29135843a3a5 WatchSource:0}: Error finding container d9b3f41857ed89e4f00c922ba85ee7a7fa45e31e0f10133705df29135843a3a5: Status 404 returned error can't find the container with id d9b3f41857ed89e4f00c922ba85ee7a7fa45e31e0f10133705df29135843a3a5
Nov 02 09:53:24.885929 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:24.885900    4614 generic.go:356] "Generic (PLEG): container finished" podID="03857226-641c-4716-948b-fe13d3b787c3" containerID="33a1bead79e260fd5f82ce76ab1cb378743382b05eddb3d754a2013ffd6266bb" exitCode=0
Nov 02 09:53:24.886338 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:24.885956    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt" event={"ID":"03857226-641c-4716-948b-fe13d3b787c3","Type":"ContainerDied","Data":"33a1bead79e260fd5f82ce76ab1cb378743382b05eddb3d754a2013ffd6266bb"}
Nov 02 09:53:24.886338 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:24.886012    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt" event={"ID":"03857226-641c-4716-948b-fe13d3b787c3","Type":"ContainerStarted","Data":"d9b3f41857ed89e4f00c922ba85ee7a7fa45e31e0f10133705df29135843a3a5"}
Nov 02 09:53:26.042773 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:26.042745    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"
Nov 02 09:53:26.132522 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:26.132489    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03857226-641c-4716-948b-fe13d3b787c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03857226-641c-4716-948b-fe13d3b787c3-maas-api-openshift-ca\") pod \"03857226-641c-4716-948b-fe13d3b787c3\" (UID: \"03857226-641c-4716-948b-fe13d3b787c3\") "
Nov 02 09:53:26.132714 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:26.132567    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03857226-641c-4716-948b-fe13d3b787c3-kube-api-access-t4fpm\" (UniqueName: \"kubernetes.io/projected/03857226-641c-4716-948b-fe13d3b787c3-kube-api-access-t4fpm\") pod \"03857226-641c-4716-948b-fe13d3b787c3\" (UID: \"03857226-641c-4716-948b-fe13d3b787c3\") "
Nov 02 09:53:26.132714 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:26.132621    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03857226-641c-4716-948b-fe13d3b787c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03857226-641c-4716-948b-fe13d3b787c3-maas-api-xks-ca\") pod \"03857226-641c-4716-948b-fe13d3b787c3\" (UID: \"03857226-641c-4716-948b-fe13d3b787c3\") "
Nov 02 09:53:26.132966 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:26.132909    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03857226-641c-4716-948b-fe13d3b787c3-maas-api-openshift-ca" pod "03857226-641c-4716-948b-fe13d3b787c3" (UID: "03857226-641c-4716-948b-fe13d3b787c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:26.134944 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:26.133427    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03857226-641c-4716-948b-fe13d3b787c3-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:26.136369 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:26.136319    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03857226-641c-4716-948b-fe13d3b787c3-maas-api-xks-ca" pod "03857226-641c-4716-948b-fe13d3b787c3" (UID: "03857226-641c-4716-948b-fe13d3b787c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:53:26.136759 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:26.136724    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03857226-641c-4716-948b-fe13d3b787c3-kube-api-access-t4fpm" pod "03857226-641c-4716-948b-fe13d3b787c3" (UID: "03857226-641c-4716-948b-fe13d3b787c3"). InnerVolumeSpecName "kube-api-access-t4fpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:26.259520 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:26.259190    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03857226-641c-4716-948b-fe13d3b787c3-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:26.259520 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:26.259231    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4fpm\" (UniqueName: \"kubernetes.io/projected/03857226-641c-4716-948b-fe13d3b787c3-kube-api-access-t4fpm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:26.924378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:26.924084    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt" event={"ID":"03857226-641c-4716-948b-fe13d3b787c3","Type":"ContainerDied","Data":"d9b3f41857ed89e4f00c922ba85ee7a7fa45e31e0f10133705df29135843a3a5"}
Nov 02 09:53:26.928373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:26.925479    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"
Nov 02 09:53:26.928373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:26.926658    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9b3f41857ed89e4f00c922ba85ee7a7fa45e31e0f10133705df29135843a3a5"
Nov 02 09:53:35.514664 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.514631    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"]
Nov 02 09:53:35.515564 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.515158    4614 cpu_manager.go:414] "containerMap: removing container" podUID="03857226-641c-4716-948b-fe13d3b787c3" containerName="revoke-keys"
Nov 02 09:53:35.515564 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.515171    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03857226-641c-4716-948b-fe13d3b787c3" containerName="revoke-keys"
Nov 02 09:53:35.515564 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.515265    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac06621-1bf7-4939-91bc-c33c0cda61a6" containerName="istio-proxy"
Nov 02 09:53:35.515564 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.515275    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="03857226-641c-4716-948b-fe13d3b787c3" containerName="revoke-keys"
Nov 02 09:53:35.517834 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.517814    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"
Nov 02 09:53:35.527130 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.527104    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:53:35.527130 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.527149    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:53:35.529344 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.529319    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"]
Nov 02 09:53:35.576451 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.576348    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-514e2f48-p5v9j\" (UID: \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"
Nov 02 09:53:35.576693 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.576591    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-514e2f48-p5v9j\" (UID: \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"
Nov 02 09:53:35.576693 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.576618    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjbl7\" (UniqueName: \"kubernetes.io/projected/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-kube-api-access-qjbl7\") pod \"maas-api-revoke-sub-514e2f48-p5v9j\" (UID: \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"
Nov 02 09:53:35.678312 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.678274    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-514e2f48-p5v9j\" (UID: \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"
Nov 02 09:53:35.678514 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.678326    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjbl7\" (UniqueName: \"kubernetes.io/projected/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-kube-api-access-qjbl7\") pod \"maas-api-revoke-sub-514e2f48-p5v9j\" (UID: \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"
Nov 02 09:53:35.678514 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.678411    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-514e2f48-p5v9j\" (UID: \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"
Nov 02 09:53:35.679167 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.679134    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-514e2f48-p5v9j\" (UID: \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"
Nov 02 09:53:35.680764 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.680733    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-514e2f48-p5v9j\" (UID: \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"
Nov 02 09:53:35.691280 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.691252    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjbl7\" (UniqueName: \"kubernetes.io/projected/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-kube-api-access-qjbl7\") pod \"maas-api-revoke-sub-514e2f48-p5v9j\" (UID: \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"
Nov 02 09:53:35.832611 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:35.832533    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"
Nov 02 09:53:36.162942 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:53:36.162891    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338ef3fd_8640_4e22_a44d_ecb1cf4e8626.slice/crio-a10815e25d861c22922931e62cabdff6a1ddd3390895c1029ca460e4edeed0dc WatchSource:0}: Error finding container a10815e25d861c22922931e62cabdff6a1ddd3390895c1029ca460e4edeed0dc: Status 404 returned error can't find the container with id a10815e25d861c22922931e62cabdff6a1ddd3390895c1029ca460e4edeed0dc
Nov 02 09:53:36.167691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:36.167583    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"]
Nov 02 09:53:37.040421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:37.040052    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j" event={"ID":"338ef3fd-8640-4e22-a44d-ecb1cf4e8626","Type":"ContainerStarted","Data":"c85b3845959eb3f34d9c95617c348278358e337c6f0a6d2f8ec90a73a59ac144"}
Nov 02 09:53:37.040421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:37.040161    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j" event={"ID":"338ef3fd-8640-4e22-a44d-ecb1cf4e8626","Type":"ContainerStarted","Data":"a10815e25d861c22922931e62cabdff6a1ddd3390895c1029ca460e4edeed0dc"}
Nov 02 09:53:37.059962 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:37.059917    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j" podStartSLOduration=2.059904167 podStartE2EDuration="2.059904167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:37.058141456 +0000 UTC m=+1097.890799535" watchObservedRunningTime="2026-11-02 09:53:37.059904167 +0000 UTC m=+1097.892562247"
Nov 02 09:53:38.048811 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:38.048779    4614 generic.go:356] "Generic (PLEG): container finished" podID="338ef3fd-8640-4e22-a44d-ecb1cf4e8626" containerID="c85b3845959eb3f34d9c95617c348278358e337c6f0a6d2f8ec90a73a59ac144" exitCode=0
Nov 02 09:53:38.049251 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:38.048860    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j" event={"ID":"338ef3fd-8640-4e22-a44d-ecb1cf4e8626","Type":"ContainerDied","Data":"c85b3845959eb3f34d9c95617c348278358e337c6f0a6d2f8ec90a73a59ac144"}
Nov 02 09:53:39.273380 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:39.273341    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"
Nov 02 09:53:39.355025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:39.354952    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-maas-api-openshift-ca\") pod \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\" (UID: \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\") "
Nov 02 09:53:39.355171 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:39.355058    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-kube-api-access-qjbl7\" (UniqueName: \"kubernetes.io/projected/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-kube-api-access-qjbl7\") pod \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\" (UID: \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\") "
Nov 02 09:53:39.355171 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:39.355116    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-maas-api-xks-ca\") pod \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\" (UID: \"338ef3fd-8640-4e22-a44d-ecb1cf4e8626\") "
Nov 02 09:53:39.355393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:39.355340    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-maas-api-openshift-ca" pod "338ef3fd-8640-4e22-a44d-ecb1cf4e8626" (UID: "338ef3fd-8640-4e22-a44d-ecb1cf4e8626"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:39.355517 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:39.355460    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:39.357090 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:39.357069    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-maas-api-xks-ca" pod "338ef3fd-8640-4e22-a44d-ecb1cf4e8626" (UID: "338ef3fd-8640-4e22-a44d-ecb1cf4e8626"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:53:39.357161 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:39.357139    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-kube-api-access-qjbl7" pod "338ef3fd-8640-4e22-a44d-ecb1cf4e8626" (UID: "338ef3fd-8640-4e22-a44d-ecb1cf4e8626"). InnerVolumeSpecName "kube-api-access-qjbl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:39.456814 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:39.456787    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjbl7\" (UniqueName: \"kubernetes.io/projected/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-kube-api-access-qjbl7\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:39.456814 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:39.456811    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/338ef3fd-8640-4e22-a44d-ecb1cf4e8626-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:39.636880 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:53:39.636791    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:53:40.070195 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:40.070161    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"
Nov 02 09:53:40.070195 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:40.070169    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j" event={"ID":"338ef3fd-8640-4e22-a44d-ecb1cf4e8626","Type":"ContainerDied","Data":"a10815e25d861c22922931e62cabdff6a1ddd3390895c1029ca460e4edeed0dc"}
Nov 02 09:53:40.070416 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:40.070264    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a10815e25d861c22922931e62cabdff6a1ddd3390895c1029ca460e4edeed0dc"
Nov 02 09:53:47.366219 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.366144    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Nov 02 09:53:47.411141 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.410463    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:53:47.457136 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.454453    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"]
Nov 02 09:53:47.457136 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.455287    4614 cpu_manager.go:414] "containerMap: removing container" podUID="338ef3fd-8640-4e22-a44d-ecb1cf4e8626" containerName="revoke-keys"
Nov 02 09:53:47.457136 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.455307    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338ef3fd-8640-4e22-a44d-ecb1cf4e8626" containerName="revoke-keys"
Nov 02 09:53:47.457136 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.455458    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="338ef3fd-8640-4e22-a44d-ecb1cf4e8626" containerName="revoke-keys"
Nov 02 09:53:47.459638 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.459614    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.472382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.462197    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"]
Nov 02 09:53:47.521413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.519946    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-credential-socket\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.521413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.520032    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-envoy\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.521413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.520121    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-workload-certs\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.521413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.520179    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-token\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.521413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.520247    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-data\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.521413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.520288    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-podinfo\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.521413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.520318    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-workload-socket\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.521413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.520391    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istiod-ca-cert\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.521413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.520447    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d68nq\" (UniqueName: \"kubernetes.io/projected/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-kube-api-access-d68nq\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.567739 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.567706    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-ced1b7-openshift-default-dockercfg-ww5mb\""
Nov 02 09:53:47.574386 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.567957    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 02 09:53:47.624182 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.624082    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d68nq\" (UniqueName: \"kubernetes.io/projected/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-kube-api-access-d68nq\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.624548 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.624525    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-credential-socket\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.624712 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.624694    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-envoy\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.624870 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.624857    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-workload-certs\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.624985 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.624972    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-token\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.625118 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.625105    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-data\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.625232 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.625220    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-podinfo\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.625334 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.625320    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-workload-socket\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.625486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.625471    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istiod-ca-cert\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.626438 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.626414    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istiod-ca-cert\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.626849 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.626829    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-credential-socket\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.628009 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.627987    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-data\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.633385 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.628745    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-workload-certs\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.633839 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.633818    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-workload-socket\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.661384 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.654700    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-podinfo\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.676388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.673118    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-envoy\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.677088 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.677061    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-token\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:47.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:47.693240    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d68nq\" (UniqueName: \"kubernetes.io/projected/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-kube-api-access-d68nq\") pod \"e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") " pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:48.028078 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:48.022974    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:48.425916 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:48.425869    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:53:48.444572 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:48.440179    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:53:48.528505 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:48.528455    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:53:48.578491 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:48.578442    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:53:48.596436 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:48.581835    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:53:48.640478 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:48.640404    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:53:48.808742 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:48.758792    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 09:53:48.808742 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:48.784456    4614 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:53:49.250175 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:49.249071    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:53:49.358397 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:49.358271    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Nov 02 09:53:49.542483 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:49.542427    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"]
Nov 02 09:53:49.618996 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:53:49.615434    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebc2cab_97a1_4a35_8f0a_a3d3ea062eb3.slice/crio-97459a5bf72aead6deb8caf02594b315c2bc9ee17fcda2c0434709e1c5f99250 WatchSource:0}: Error finding container 97459a5bf72aead6deb8caf02594b315c2bc9ee17fcda2c0434709e1c5f99250: Status 404 returned error can't find the container with id 97459a5bf72aead6deb8caf02594b315c2bc9ee17fcda2c0434709e1c5f99250
Nov 02 09:53:49.634035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:49.633274    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:53:49.634035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:49.633370    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:53:49.634035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:49.633416    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:53:50.287884 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:50.287838    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:53:50.288052 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:50.287935    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:53:50.634483 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:50.634386    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" event={"ID":"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3","Type":"ContainerStarted","Data":"2b3696c84c9a9a201482b7ed6b58cf3e482b26e6c47ee6fc8e0d40cdd047e963"}
Nov 02 09:53:50.634483 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:50.634455    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" event={"ID":"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3","Type":"ContainerStarted","Data":"97459a5bf72aead6deb8caf02594b315c2bc9ee17fcda2c0434709e1c5f99250"}
Nov 02 09:53:50.658808 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:50.658749    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" podStartSLOduration=4.65870694 podStartE2EDuration="4.65870694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:50.657727957 +0000 UTC m=+1111.490386037" watchObservedRunningTime="2026-11-02 09:53:50.65870694 +0000 UTC m=+1111.491365021"
Nov 02 09:53:51.025748 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.025714    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:51.036618 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.036585    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:51.681449 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.681405    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:51.690331 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.690053    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:53:51.696057 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.696027    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"]
Nov 02 09:53:51.732989 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.732961    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"
Nov 02 09:53:51.738136 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.738104    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"]
Nov 02 09:53:51.744383 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.739000    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:53:51.784503 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.759491    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:53:51.785845 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.785816    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c3e1453-7v89m\" (UID: \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"
Nov 02 09:53:51.786143 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.786122    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c3e1453-7v89m\" (UID: \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"
Nov 02 09:53:51.786296 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.786280    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqmnv\" (UniqueName: \"kubernetes.io/projected/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-kube-api-access-mqmnv\") pod \"maas-api-revoke-sub-9c3e1453-7v89m\" (UID: \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"
Nov 02 09:53:51.887218 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.887182    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c3e1453-7v89m\" (UID: \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"
Nov 02 09:53:51.887417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.887238    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c3e1453-7v89m\" (UID: \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"
Nov 02 09:53:51.887417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.887265    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmnv\" (UniqueName: \"kubernetes.io/projected/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-kube-api-access-mqmnv\") pod \"maas-api-revoke-sub-9c3e1453-7v89m\" (UID: \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"
Nov 02 09:53:51.887974 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.887950    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c3e1453-7v89m\" (UID: \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"
Nov 02 09:53:51.889577 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.889551    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c3e1453-7v89m\" (UID: \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"
Nov 02 09:53:51.900033 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:51.900011    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqmnv\" (UniqueName: \"kubernetes.io/projected/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-kube-api-access-mqmnv\") pod \"maas-api-revoke-sub-9c3e1453-7v89m\" (UID: \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"
Nov 02 09:53:52.107821 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:52.107785    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"
Nov 02 09:53:52.577834 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:52.577801    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"]
Nov 02 09:53:52.578542 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:53:52.578516    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b8d319_6b2e_42c2_b7c5_0ead72698cc2.slice/crio-8cabccb71581b1aa0b683ded74cb107e954c7e119e30434b24aa3d9fc8426ae8 WatchSource:0}: Error finding container 8cabccb71581b1aa0b683ded74cb107e954c7e119e30434b24aa3d9fc8426ae8: Status 404 returned error can't find the container with id 8cabccb71581b1aa0b683ded74cb107e954c7e119e30434b24aa3d9fc8426ae8
Nov 02 09:53:52.674184 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:52.674140    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m" event={"ID":"53b8d319-6b2e-42c2-b7c5-0ead72698cc2","Type":"ContainerStarted","Data":"8cabccb71581b1aa0b683ded74cb107e954c7e119e30434b24aa3d9fc8426ae8"}
Nov 02 09:53:53.682339 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:53.682291    4614 generic.go:356] "Generic (PLEG): container finished" podID="53b8d319-6b2e-42c2-b7c5-0ead72698cc2" containerID="c52fab50d15f562885cd202a2e60682600f0d047142403c73d03e834202473e6" exitCode=0
Nov 02 09:53:53.682819 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:53.682401    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m" event={"ID":"53b8d319-6b2e-42c2-b7c5-0ead72698cc2","Type":"ContainerDied","Data":"c52fab50d15f562885cd202a2e60682600f0d047142403c73d03e834202473e6"}
Nov 02 09:53:54.447922 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:54.447886    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"]
Nov 02 09:53:54.452021 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:54.452001    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"
Nov 02 09:53:54.461327 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:54.461299    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-ced1b7\""
Nov 02 09:53:54.461516 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:54.461412    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-ced1b7-dockercfg-vhvlk\""
Nov 02 09:53:54.466850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:54.466825    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"]
Nov 02 09:53:54.491476 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:54.491430    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chh2k\" (UniqueName: \"kubernetes.io/projected/0507afd5-3cdb-4432-8c5e-3322dddd7ed9-kube-api-access-chh2k\") pod \"payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8\" (UID: \"0507afd5-3cdb-4432-8c5e-3322dddd7ed9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"
Nov 02 09:53:54.491640 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:54.491620    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0507afd5-3cdb-4432-8c5e-3322dddd7ed9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8\" (UID: \"0507afd5-3cdb-4432-8c5e-3322dddd7ed9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"
Nov 02 09:53:54.608375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:54.606480    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chh2k\" (UniqueName: \"kubernetes.io/projected/0507afd5-3cdb-4432-8c5e-3322dddd7ed9-kube-api-access-chh2k\") pod \"payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8\" (UID: \"0507afd5-3cdb-4432-8c5e-3322dddd7ed9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"
Nov 02 09:53:54.608375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:54.606639    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0507afd5-3cdb-4432-8c5e-3322dddd7ed9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8\" (UID: \"0507afd5-3cdb-4432-8c5e-3322dddd7ed9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"
Nov 02 09:53:54.638038 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:54.638002    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0507afd5-3cdb-4432-8c5e-3322dddd7ed9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8\" (UID: \"0507afd5-3cdb-4432-8c5e-3322dddd7ed9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"
Nov 02 09:53:54.640865 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:54.640839    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chh2k\" (UniqueName: \"kubernetes.io/projected/0507afd5-3cdb-4432-8c5e-3322dddd7ed9-kube-api-access-chh2k\") pod \"payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8\" (UID: \"0507afd5-3cdb-4432-8c5e-3322dddd7ed9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"
Nov 02 09:53:54.770894 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:54.770853    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"
Nov 02 09:53:55.132598 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.132566    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"
Nov 02 09:53:55.204105 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.202742    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"]
Nov 02 09:53:55.208474 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:53:55.208448    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0507afd5_3cdb_4432_8c5e_3322dddd7ed9.slice/crio-50d316bdc75eebd62b8eb5d477af01e7a2d5618ca4fd2457d163619713f12934 WatchSource:0}: Error finding container 50d316bdc75eebd62b8eb5d477af01e7a2d5618ca4fd2457d163619713f12934: Status 404 returned error can't find the container with id 50d316bdc75eebd62b8eb5d477af01e7a2d5618ca4fd2457d163619713f12934
Nov 02 09:53:55.222861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.222838    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-maas-api-openshift-ca\") pod \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\" (UID: \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\") "
Nov 02 09:53:55.222958 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.222943    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-kube-api-access-mqmnv\" (UniqueName: \"kubernetes.io/projected/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-kube-api-access-mqmnv\") pod \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\" (UID: \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\") "
Nov 02 09:53:55.223178 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.223125    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-maas-api-xks-ca\") pod \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\" (UID: \"53b8d319-6b2e-42c2-b7c5-0ead72698cc2\") "
Nov 02 09:53:55.223605 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.223189    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-maas-api-openshift-ca" pod "53b8d319-6b2e-42c2-b7c5-0ead72698cc2" (UID: "53b8d319-6b2e-42c2-b7c5-0ead72698cc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:53:55.223760 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.223741    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:55.227746 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.227725    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-kube-api-access-mqmnv" pod "53b8d319-6b2e-42c2-b7c5-0ead72698cc2" (UID: "53b8d319-6b2e-42c2-b7c5-0ead72698cc2"). InnerVolumeSpecName "kube-api-access-mqmnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:53:55.229634 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.229601    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-maas-api-xks-ca" pod "53b8d319-6b2e-42c2-b7c5-0ead72698cc2" (UID: "53b8d319-6b2e-42c2-b7c5-0ead72698cc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:53:55.325146 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.325122    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:55.325146 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.325147    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqmnv\" (UniqueName: \"kubernetes.io/projected/53b8d319-6b2e-42c2-b7c5-0ead72698cc2-kube-api-access-mqmnv\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:53:55.812429 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.812339    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8" event={"ID":"0507afd5-3cdb-4432-8c5e-3322dddd7ed9","Type":"ContainerStarted","Data":"507cd33d897c3499a3078acb6d2bd7a1ef02ce8c96cffab573fa1b6a2251e465"}
Nov 02 09:53:55.812932 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.812882    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8" event={"ID":"0507afd5-3cdb-4432-8c5e-3322dddd7ed9","Type":"ContainerStarted","Data":"50d316bdc75eebd62b8eb5d477af01e7a2d5618ca4fd2457d163619713f12934"}
Nov 02 09:53:55.812996 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.812953    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"
Nov 02 09:53:55.814591 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.814569    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"
Nov 02 09:53:55.814591 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.814575    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m" event={"ID":"53b8d319-6b2e-42c2-b7c5-0ead72698cc2","Type":"ContainerDied","Data":"8cabccb71581b1aa0b683ded74cb107e954c7e119e30434b24aa3d9fc8426ae8"}
Nov 02 09:53:55.814899 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.814607    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cabccb71581b1aa0b683ded74cb107e954c7e119e30434b24aa3d9fc8426ae8"
Nov 02 09:53:55.831608 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:53:55.831550    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8" podStartSLOduration=1.831532613 podStartE2EDuration="1.831532613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:53:55.829780917 +0000 UTC m=+1116.662439138" watchObservedRunningTime="2026-11-02 09:53:55.831532613 +0000 UTC m=+1116.664190706"
Nov 02 09:54:04.395523 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.388262    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"]
Nov 02 09:54:04.395523 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.389092    4614 cpu_manager.go:414] "containerMap: removing container" podUID="53b8d319-6b2e-42c2-b7c5-0ead72698cc2" containerName="revoke-keys"
Nov 02 09:54:04.395523 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.389111    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b8d319-6b2e-42c2-b7c5-0ead72698cc2" containerName="revoke-keys"
Nov 02 09:54:04.395523 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.389262    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b8d319-6b2e-42c2-b7c5-0ead72698cc2" containerName="revoke-keys"
Nov 02 09:54:04.396065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.395608    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"]
Nov 02 09:54:04.396065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.395766    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"
Nov 02 09:54:04.430658 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.430631    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:54:04.431005 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.430985    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:54:04.534366 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.534321    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224ccb7a-946a-4bf6-90a0-4cf63b62be04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccef70b8-lq95b\" (UID: \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"
Nov 02 09:54:04.534579 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.534469    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gxg5\" (UniqueName: \"kubernetes.io/projected/224ccb7a-946a-4bf6-90a0-4cf63b62be04-kube-api-access-5gxg5\") pod \"maas-api-revoke-sub-ccef70b8-lq95b\" (UID: \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"
Nov 02 09:54:04.534971 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.534946    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224ccb7a-946a-4bf6-90a0-4cf63b62be04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccef70b8-lq95b\" (UID: \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"
Nov 02 09:54:04.636007 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.635968    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gxg5\" (UniqueName: \"kubernetes.io/projected/224ccb7a-946a-4bf6-90a0-4cf63b62be04-kube-api-access-5gxg5\") pod \"maas-api-revoke-sub-ccef70b8-lq95b\" (UID: \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"
Nov 02 09:54:04.636155 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.636094    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224ccb7a-946a-4bf6-90a0-4cf63b62be04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccef70b8-lq95b\" (UID: \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"
Nov 02 09:54:04.636230 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.636214    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224ccb7a-946a-4bf6-90a0-4cf63b62be04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccef70b8-lq95b\" (UID: \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"
Nov 02 09:54:04.636822 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.636794    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224ccb7a-946a-4bf6-90a0-4cf63b62be04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccef70b8-lq95b\" (UID: \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"
Nov 02 09:54:04.638723 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.638696    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224ccb7a-946a-4bf6-90a0-4cf63b62be04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccef70b8-lq95b\" (UID: \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"
Nov 02 09:54:04.649055 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.648989    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gxg5\" (UniqueName: \"kubernetes.io/projected/224ccb7a-946a-4bf6-90a0-4cf63b62be04-kube-api-access-5gxg5\") pod \"maas-api-revoke-sub-ccef70b8-lq95b\" (UID: \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"
Nov 02 09:54:04.743467 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:04.743437    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"
Nov 02 09:54:05.028134 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:05.028102    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"]
Nov 02 09:54:05.032290 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:05.032250    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"]
Nov 02 09:54:05.032804 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:54:05.032776    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224ccb7a_946a_4bf6_90a0_4cf63b62be04.slice/crio-a4174c6008525f4cd80780a1e1d2306bc92e543812f104aab17aafeeb3158f78 WatchSource:0}: Error finding container a4174c6008525f4cd80780a1e1d2306bc92e543812f104aab17aafeeb3158f78: Status 404 returned error can't find the container with id a4174c6008525f4cd80780a1e1d2306bc92e543812f104aab17aafeeb3158f78
Nov 02 09:54:05.035824 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:05.035804    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be24fe9a-f8g4c"]
Nov 02 09:54:05.317577 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:05.317494    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7afacc90-28a8-43ff-9629-992ee73f304b" path="/var/lib/kubelet/pods/7afacc90-28a8-43ff-9629-992ee73f304b/volumes"
Nov 02 09:54:05.965679 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:05.965642    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b" event={"ID":"224ccb7a-946a-4bf6-90a0-4cf63b62be04","Type":"ContainerStarted","Data":"ec8b0e128e54dc016778feb3574c4fb917b48a074c801d8cdacf14e3560bde7d"}
Nov 02 09:54:05.966124 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:05.965694    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b" event={"ID":"224ccb7a-946a-4bf6-90a0-4cf63b62be04","Type":"ContainerStarted","Data":"a4174c6008525f4cd80780a1e1d2306bc92e543812f104aab17aafeeb3158f78"}
Nov 02 09:54:05.984415 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:05.984344    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b" podStartSLOduration=1.9843302760000001 podStartE2EDuration="1.984330276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:05.983135702 +0000 UTC m=+1126.815793796" watchObservedRunningTime="2026-11-02 09:54:05.984330276 +0000 UTC m=+1126.816988356"
Nov 02 09:54:06.067461 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:54:06.067423    4614 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224ccb7a_946a_4bf6_90a0_4cf63b62be04.slice/crio-conmon-ec8b0e128e54dc016778feb3574c4fb917b48a074c801d8cdacf14e3560bde7d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224ccb7a_946a_4bf6_90a0_4cf63b62be04.slice/crio-ec8b0e128e54dc016778feb3574c4fb917b48a074c801d8cdacf14e3560bde7d.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 09:54:06.067461 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:54:06.067447    4614 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224ccb7a_946a_4bf6_90a0_4cf63b62be04.slice/crio-ec8b0e128e54dc016778feb3574c4fb917b48a074c801d8cdacf14e3560bde7d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224ccb7a_946a_4bf6_90a0_4cf63b62be04.slice/crio-conmon-ec8b0e128e54dc016778feb3574c4fb917b48a074c801d8cdacf14e3560bde7d.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 09:54:06.824386 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:06.824328    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"
Nov 02 09:54:07.015398 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:07.003076    4614 generic.go:356] "Generic (PLEG): container finished" podID="224ccb7a-946a-4bf6-90a0-4cf63b62be04" containerID="ec8b0e128e54dc016778feb3574c4fb917b48a074c801d8cdacf14e3560bde7d" exitCode=0
Nov 02 09:54:07.015398 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:07.003325    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b" event={"ID":"224ccb7a-946a-4bf6-90a0-4cf63b62be04","Type":"ContainerDied","Data":"ec8b0e128e54dc016778feb3574c4fb917b48a074c801d8cdacf14e3560bde7d"}
Nov 02 09:54:07.980699 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:07.980663    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"]
Nov 02 09:54:07.985335 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:07.985313    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:07.997388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:07.997361    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"]
Nov 02 09:54:07.999456 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:07.999430    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:07.999593 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:07.999459    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b039b3-0a79-43d3-bd83-347582902302-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:07.999655 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:07.999602    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b039b3-0a79-43d3-bd83-347582902302-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:07.999655 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:07.999633    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:07.999761 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:07.999666    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:07.999761 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:07.999703    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:07.999924 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:07.999898    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:07.999988 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:07.999947    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b039b3-0a79-43d3-bd83-347582902302-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.000128 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.000109    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvqv\" (UniqueName: \"kubernetes.io/projected/48b039b3-0a79-43d3-bd83-347582902302-kube-api-access-5lvqv\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.124587 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.124552    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b039b3-0a79-43d3-bd83-347582902302-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.126063 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.126042    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvqv\" (UniqueName: \"kubernetes.io/projected/48b039b3-0a79-43d3-bd83-347582902302-kube-api-access-5lvqv\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.126698 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.126678    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.127188 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.127169    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b039b3-0a79-43d3-bd83-347582902302-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.127673 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.127632    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b039b3-0a79-43d3-bd83-347582902302-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.128244 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.128222    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.128725 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.128710    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.129076 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.129064    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.129404 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.129392    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.129777 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.129019    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-istio-data\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.130239 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.125930    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b039b3-0a79-43d3-bd83-347582902302-istiod-ca-cert\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.130346 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.128602    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-credential-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.130465 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.129307    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-workload-certs\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.130554 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.127118    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-workload-socket\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.130637 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.130202    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b039b3-0a79-43d3-bd83-347582902302-istio-podinfo\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.131293 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.131279    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-istio-envoy\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.139681 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.139658    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvqv\" (UniqueName: \"kubernetes.io/projected/48b039b3-0a79-43d3-bd83-347582902302-kube-api-access-5lvqv\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.140532 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.140510    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b039b3-0a79-43d3-bd83-347582902302-istio-token\") pod \"e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") " pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.238345 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.238265    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"
Nov 02 09:54:08.299952 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.299924    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:08.331572 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.331543    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/224ccb7a-946a-4bf6-90a0-4cf63b62be04-kube-api-access-5gxg5\" (UniqueName: \"kubernetes.io/projected/224ccb7a-946a-4bf6-90a0-4cf63b62be04-kube-api-access-5gxg5\") pod \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\" (UID: \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\") "
Nov 02 09:54:08.331726 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.331708    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/224ccb7a-946a-4bf6-90a0-4cf63b62be04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224ccb7a-946a-4bf6-90a0-4cf63b62be04-maas-api-xks-ca\") pod \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\" (UID: \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\") "
Nov 02 09:54:08.331811 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.331796    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/224ccb7a-946a-4bf6-90a0-4cf63b62be04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224ccb7a-946a-4bf6-90a0-4cf63b62be04-maas-api-openshift-ca\") pod \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\" (UID: \"224ccb7a-946a-4bf6-90a0-4cf63b62be04\") "
Nov 02 09:54:08.332205 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.332176    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224ccb7a-946a-4bf6-90a0-4cf63b62be04-maas-api-openshift-ca" pod "224ccb7a-946a-4bf6-90a0-4cf63b62be04" (UID: "224ccb7a-946a-4bf6-90a0-4cf63b62be04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:08.332407 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.332389    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224ccb7a-946a-4bf6-90a0-4cf63b62be04-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:08.333658 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.333636    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224ccb7a-946a-4bf6-90a0-4cf63b62be04-kube-api-access-5gxg5" pod "224ccb7a-946a-4bf6-90a0-4cf63b62be04" (UID: "224ccb7a-946a-4bf6-90a0-4cf63b62be04"). InnerVolumeSpecName "kube-api-access-5gxg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:08.333741 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.333727    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224ccb7a-946a-4bf6-90a0-4cf63b62be04-maas-api-xks-ca" pod "224ccb7a-946a-4bf6-90a0-4cf63b62be04" (UID: "224ccb7a-946a-4bf6-90a0-4cf63b62be04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:54:08.433818 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.433788    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224ccb7a-946a-4bf6-90a0-4cf63b62be04-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:08.433818 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.433815    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gxg5\" (UniqueName: \"kubernetes.io/projected/224ccb7a-946a-4bf6-90a0-4cf63b62be04-kube-api-access-5gxg5\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:08.586895 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.586869    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"]
Nov 02 09:54:08.587428 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:54:08.587398    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b039b3_0a79_43d3_bd83_347582902302.slice/crio-47a6c0cf1bcddf2dd5f53ffd23a1b2f24d8dd82989a362d23124a842c534311e WatchSource:0}: Error finding container 47a6c0cf1bcddf2dd5f53ffd23a1b2f24d8dd82989a362d23124a842c534311e: Status 404 returned error can't find the container with id 47a6c0cf1bcddf2dd5f53ffd23a1b2f24d8dd82989a362d23124a842c534311e
Nov 02 09:54:08.590003 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.589967    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:08.590117 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.590044    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:08.590117 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:08.590088    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:09.052480 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:09.052261    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" event={"ID":"48b039b3-0a79-43d3-bd83-347582902302","Type":"ContainerStarted","Data":"2ef72950187a7cfb442fb2654d2188de5b3d961ecee9dfef68725fa28b1f9241"}
Nov 02 09:54:09.052480 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:09.052324    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" event={"ID":"48b039b3-0a79-43d3-bd83-347582902302","Type":"ContainerStarted","Data":"47a6c0cf1bcddf2dd5f53ffd23a1b2f24d8dd82989a362d23124a842c534311e"}
Nov 02 09:54:09.053824 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:09.053797    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"
Nov 02 09:54:09.053937 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:09.053799    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b" event={"ID":"224ccb7a-946a-4bf6-90a0-4cf63b62be04","Type":"ContainerDied","Data":"a4174c6008525f4cd80780a1e1d2306bc92e543812f104aab17aafeeb3158f78"}
Nov 02 09:54:09.053937 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:09.053916    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4174c6008525f4cd80780a1e1d2306bc92e543812f104aab17aafeeb3158f78"
Nov 02 09:54:09.077287 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:09.077236    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" podStartSLOduration=2.077223294 podStartE2EDuration="2.077223294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:09.074633216 +0000 UTC m=+1129.907291298" watchObservedRunningTime="2026-11-02 09:54:09.077223294 +0000 UTC m=+1129.909881373"
Nov 02 09:54:09.300677 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:09.300642    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:09.302338 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:09.302310    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 09:54:09.302542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:09.302386    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" podUID="48b039b3-0a79-43d3-bd83-347582902302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 09:54:09.681431 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:54:09.681170    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:54:10.310375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:10.300664    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 09:54:10.310375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:10.300720    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" podUID="48b039b3-0a79-43d3-bd83-347582902302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 09:54:11.300611 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.300576    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 09:54:11.300785 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.300626    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" podUID="48b039b3-0a79-43d3-bd83-347582902302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 09:54:11.324290 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.324256    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"]
Nov 02 09:54:11.325093 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.325071    4614 cpu_manager.go:414] "containerMap: removing container" podUID="224ccb7a-946a-4bf6-90a0-4cf63b62be04" containerName="revoke-keys"
Nov 02 09:54:11.325165 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.325097    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="224ccb7a-946a-4bf6-90a0-4cf63b62be04" containerName="revoke-keys"
Nov 02 09:54:11.325279 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.325266    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="224ccb7a-946a-4bf6-90a0-4cf63b62be04" containerName="revoke-keys"
Nov 02 09:54:11.360700 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.360671    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:11.363799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.363769    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"]
Nov 02 09:54:11.364906 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.364885    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9cbd331c-serving-cert\""
Nov 02 09:54:11.417897 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.417862    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b72964-98f0-44cb-9b75-6e79e4c8a17d-openshift-service-ca\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:11.418070 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.417915    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b72964-98f0-44cb-9b75-6e79e4c8a17d-metrics-certs\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:11.418154 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.418075    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b72964-98f0-44cb-9b75-6e79e4c8a17d-maas-api-tls\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:11.418217 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.418183    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx7cl\" (UniqueName: \"kubernetes.io/projected/44b72964-98f0-44cb-9b75-6e79e4c8a17d-kube-api-access-nx7cl\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:11.522090 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.522046    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b72964-98f0-44cb-9b75-6e79e4c8a17d-maas-api-tls\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:11.522276 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.522207    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx7cl\" (UniqueName: \"kubernetes.io/projected/44b72964-98f0-44cb-9b75-6e79e4c8a17d-kube-api-access-nx7cl\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:11.522340 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.522308    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b72964-98f0-44cb-9b75-6e79e4c8a17d-openshift-service-ca\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:11.522461 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.522367    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b72964-98f0-44cb-9b75-6e79e4c8a17d-metrics-certs\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:11.565426 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.547841    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b72964-98f0-44cb-9b75-6e79e4c8a17d-openshift-service-ca\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:11.565426 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.548525    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b72964-98f0-44cb-9b75-6e79e4c8a17d-metrics-certs\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:11.565426 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.551622    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b72964-98f0-44cb-9b75-6e79e4c8a17d-maas-api-tls\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:11.569641 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.569590    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx7cl\" (UniqueName: \"kubernetes.io/projected/44b72964-98f0-44cb-9b75-6e79e4c8a17d-kube-api-access-nx7cl\") pod \"maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:11.680755 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:11.680721    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:12.033694 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.033643    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"]
Nov 02 09:54:12.047211 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.047175    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"]
Nov 02 09:54:12.209003 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.208969    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"]
Nov 02 09:54:12.209332 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.209310    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.226443 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.226410    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ktc9\" (UniqueName: \"kubernetes.io/projected/7a55948d-7d73-4560-85fa-88cb221e3e9d-kube-api-access-5ktc9\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.226673 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.226652    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.226815 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.226798    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.226958 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.226942    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.227064 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.227050    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.227242 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.227227    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a55948d-7d73-4560-85fa-88cb221e3e9d-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.227458 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.227442    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.227612 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.227597    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.227732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.227717    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.301969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.301881    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 09:54:12.301969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.301953    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" podUID="48b039b3-0a79-43d3-bd83-347582902302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 09:54:12.356410 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.349139    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ktc9\" (UniqueName: \"kubernetes.io/projected/7a55948d-7d73-4560-85fa-88cb221e3e9d-kube-api-access-5ktc9\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.356410 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.349210    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.356410 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.349240    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.356410 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.349289    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.356410 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.349323    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.358486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.357248    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a55948d-7d73-4560-85fa-88cb221e3e9d-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.358486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.357412    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.358486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.357472    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.358486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.357513    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.358486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.357690    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-envoy\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.358486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.357894    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-credential-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.358486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.357943    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-data\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.358486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.358143    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-workload-socket\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.358486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.358314    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-workload-certs\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.358982 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.358577    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a55948d-7d73-4560-85fa-88cb221e3e9d-istiod-ca-cert\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.361556 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.361534    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-podinfo\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.362431 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.362410    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ktc9\" (UniqueName: \"kubernetes.io/projected/7a55948d-7d73-4560-85fa-88cb221e3e9d-kube-api-access-5ktc9\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.365406 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.365385    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-token\") pod \"e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") " pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.557951 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.557869    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:12.959646 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.959428    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"]
Nov 02 09:54:12.969439 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.969406    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:12.969521 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.969492    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:12.969554 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:12.969540    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:13.275111 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:13.275074    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q" event={"ID":"7a55948d-7d73-4560-85fa-88cb221e3e9d","Type":"ContainerStarted","Data":"c84d182b1e2f5544e44f0607d6fe3eaa3a6bc4ed3b835b78c33460d15ba3f844"}
Nov 02 09:54:13.275318 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:13.275128    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q" event={"ID":"7a55948d-7d73-4560-85fa-88cb221e3e9d","Type":"ContainerStarted","Data":"7853ac9dcf282b208100b783ce18215e70e870bcb8af9e8378a8d834fed5ed94"}
Nov 02 09:54:13.277724 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:13.277696    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn" event={"ID":"44b72964-98f0-44cb-9b75-6e79e4c8a17d","Type":"ContainerStarted","Data":"d6d1600000870604bdbc12748bc2c10b2eeafa927498ba543ec10f7b93d2abee"}
Nov 02 09:54:13.277869 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:13.277739    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn" event={"ID":"44b72964-98f0-44cb-9b75-6e79e4c8a17d","Type":"ContainerStarted","Data":"e2668755028d218a19d3fb8ef72a8d7180e0608e8a2a6408ecd5d096de91b27a"}
Nov 02 09:54:13.277980 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:13.277959    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:13.300996 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:13.300962    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 09:54:13.301138 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:13.301005    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" podUID="48b039b3-0a79-43d3-bd83-347582902302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 09:54:13.309144 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:13.309066    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q" podStartSLOduration=1.309050011 podStartE2EDuration="1.309050011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:13.305982862 +0000 UTC m=+1134.138640977" watchObservedRunningTime="2026-11-02 09:54:13.309050011 +0000 UTC m=+1134.141708094"
Nov 02 09:54:13.346439 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:13.346373    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn" podStartSLOduration=1.7281072960000001 podStartE2EDuration="2.346333533s" totalImagesPullingTime="618.226237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:13.344563977 +0000 UTC m=+1134.177222060" watchObservedRunningTime="2026-11-02 09:54:13.346333533 +0000 UTC m=+1134.178991613"
Nov 02 09:54:13.565018 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:13.564937    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:14.300703 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.300661    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 09:54:14.300937 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.300916    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" podUID="48b039b3-0a79-43d3-bd83-347582902302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 09:54:14.571712 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.571288    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 09:54:14.571712 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.571382    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q" podUID="7a55948d-7d73-4560-85fa-88cb221e3e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 09:54:14.628449 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.628411    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"]
Nov 02 09:54:14.657038 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.657006    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:14.680343 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.680301    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"]
Nov 02 09:54:14.752388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.748979    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-ced1b7\"/\"openshift-service-ca.crt\""
Nov 02 09:54:14.752388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.749164    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-ced1b7\"/\"default-dockercfg-nz8pw\""
Nov 02 09:54:14.752388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.749283    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-ced1b7\"/\"autores-model-ced1b7-kserve-self-signed-certs\""
Nov 02 09:54:14.752388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.749391    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-ced1b7\"/\"kube-root-ca.crt\""
Nov 02 09:54:14.752388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.750698    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-tmp-dir\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:14.752388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.750762    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-dshm\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:14.752388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.750795    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whwq7\" (UniqueName: \"kubernetes.io/projected/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-kube-api-access-whwq7\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:14.752388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.750829    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-home\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:14.752388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.750854    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-model-cache\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:14.752388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:14.750983    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-tls-certs\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:15.054455 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.041586    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-tls-certs\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:15.055101 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.055054    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-tmp-dir\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:15.055969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.055941    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-dshm\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:15.086513 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.080971    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whwq7\" (UniqueName: \"kubernetes.io/projected/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-kube-api-access-whwq7\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:15.130246 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.130202    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-home\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:15.130543 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.130498    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-home\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:15.130712 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.130696    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-model-cache\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:15.134623 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.130829    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-dshm\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:15.138376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.131073    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-model-cache\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:15.138528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.134430    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-tmp-dir\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:15.161495 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.152125    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whwq7\" (UniqueName: \"kubernetes.io/projected/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-kube-api-access-whwq7\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:15.164582 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.160162    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-tls-certs\") pod \"autores-model-ced1b7-kserve-685cc8796f-c7k6t\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") " pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:15.300795 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.300762    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 09:54:15.300981 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.300821    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" podUID="48b039b3-0a79-43d3-bd83-347582902302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 09:54:15.402428 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.398611    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:15.558982 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.558950    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 09:54:15.572801 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:15.572743    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q" podUID="7a55948d-7d73-4560-85fa-88cb221e3e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 09:54:16.284506 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:16.277944    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"]
Nov 02 09:54:16.325318 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:16.324517    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 02 09:54:16.325318 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:16.324577    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" podUID="48b039b3-0a79-43d3-bd83-347582902302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 02 09:54:16.397046 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:16.392571    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t" event={"ID":"6ad89d95-c57a-47e7-ac5b-efca4018cfbd","Type":"ContainerStarted","Data":"4119e3b488879c4a56fd404f100efe57d398d2100c916eab61d70041259b3610"}
Nov 02 09:54:16.617138 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:16.583837    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 09:54:16.617138 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:16.583900    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q" podUID="7a55948d-7d73-4560-85fa-88cb221e3e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 09:54:17.564194 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:17.563737    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 09:54:17.564194 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:17.563802    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q" podUID="7a55948d-7d73-4560-85fa-88cb221e3e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 09:54:17.711671 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:17.711629    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:54:17.729059 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:17.728014    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:54:17.753488 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:17.753446    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:54:18.068452 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:18.068411    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"]
Nov 02 09:54:18.086611 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:18.086570    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2e8765b-gb75g"]
Nov 02 09:54:18.320429 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:18.318558    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:54:18.320429 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:18.318623    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" podUID="48b039b3-0a79-43d3-bd83-347582902302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:54:18.368987 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:18.367685    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:54:18.565498 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:18.563379    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 09:54:18.565498 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:18.563455    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q" podUID="7a55948d-7d73-4560-85fa-88cb221e3e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 09:54:18.729376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:18.727416    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 09:54:18.743698 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:18.729905    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:54:18.817377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:18.812843    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:54:18.821376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:18.820660    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:54:18.924379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:18.920769    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:54:19.017570 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:19.017490    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 09:54:19.324419 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:19.324323    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:54:19.420598 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:19.418261    4614 patch_prober.go:29] interesting pod/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:54:19.420598 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:19.418318    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" podUID="48b039b3-0a79-43d3-bd83-347582902302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:54:19.601530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:19.592496    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 09:54:19.601530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:19.592572    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q" podUID="7a55948d-7d73-4560-85fa-88cb221e3e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 09:54:19.614172 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:19.614126    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994b0fe2-4573-4250-929e-0cd7d12e2070" path="/var/lib/kubelet/pods/994b0fe2-4573-4250-929e-0cd7d12e2070/volumes"
Nov 02 09:54:19.614979 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:19.614950    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t" event={"ID":"6ad89d95-c57a-47e7-ac5b-efca4018cfbd","Type":"ContainerStarted","Data":"841d4201443920aedfc5b51be349d6e48372d83a5828247bfa9751ca75b01750"}
Nov 02 09:54:19.615109 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:19.615083    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:19.616081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:19.616058    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:19.617211 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:19.617191    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:54:19.705802 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:19.705747    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t" podStartSLOduration=5.705732684 podStartE2EDuration="5.705732684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:19.678462204 +0000 UTC m=+1140.511120296" watchObservedRunningTime="2026-11-02 09:54:19.705732684 +0000 UTC m=+1140.538390764"
Nov 02 09:54:19.896471 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:19.891431    4614 scope.go:122] "RemoveContainer" containerID="54ecee23272c2c550add6fc3abac5562220c83c3212c7c9d1b6cf79c88bd3997"
Nov 02 09:54:20.402483 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:20.402447    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:20.563767 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:20.563731    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 09:54:20.564096 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:20.564068    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q" podUID="7a55948d-7d73-4560-85fa-88cb221e3e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 09:54:21.562400 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:21.561944    4614 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 09:54:21.562400 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:21.562078    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q" podUID="7a55948d-7d73-4560-85fa-88cb221e3e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 09:54:22.373997 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.371403    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"]
Nov 02 09:54:22.380542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.380515    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"
Nov 02 09:54:22.383224 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.383201    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"]
Nov 02 09:54:22.388301 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.388258    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 09:54:22.389653 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.389619    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 09:54:22.506931 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.506891    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2323d4e8-45f0-4109-9c14-48293a0805d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4910eac-f5wzv\" (UID: \"2323d4e8-45f0-4109-9c14-48293a0805d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"
Nov 02 09:54:22.507120 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.507079    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2323d4e8-45f0-4109-9c14-48293a0805d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4910eac-f5wzv\" (UID: \"2323d4e8-45f0-4109-9c14-48293a0805d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"
Nov 02 09:54:22.507190 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.507174    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74zh6\" (UniqueName: \"kubernetes.io/projected/2323d4e8-45f0-4109-9c14-48293a0805d4-kube-api-access-74zh6\") pod \"maas-api-revoke-sub-a4910eac-f5wzv\" (UID: \"2323d4e8-45f0-4109-9c14-48293a0805d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"
Nov 02 09:54:22.584798 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.584765    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:22.585414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.585385    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:22.586851 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.586812    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:54:22.608569 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.608536    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2323d4e8-45f0-4109-9c14-48293a0805d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4910eac-f5wzv\" (UID: \"2323d4e8-45f0-4109-9c14-48293a0805d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"
Nov 02 09:54:22.608726 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.608701    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2323d4e8-45f0-4109-9c14-48293a0805d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4910eac-f5wzv\" (UID: \"2323d4e8-45f0-4109-9c14-48293a0805d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"
Nov 02 09:54:22.608802 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.608784    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74zh6\" (UniqueName: \"kubernetes.io/projected/2323d4e8-45f0-4109-9c14-48293a0805d4-kube-api-access-74zh6\") pod \"maas-api-revoke-sub-a4910eac-f5wzv\" (UID: \"2323d4e8-45f0-4109-9c14-48293a0805d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"
Nov 02 09:54:22.609494 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.609470    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2323d4e8-45f0-4109-9c14-48293a0805d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4910eac-f5wzv\" (UID: \"2323d4e8-45f0-4109-9c14-48293a0805d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"
Nov 02 09:54:22.611677 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.611649    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2323d4e8-45f0-4109-9c14-48293a0805d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4910eac-f5wzv\" (UID: \"2323d4e8-45f0-4109-9c14-48293a0805d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"
Nov 02 09:54:22.621923 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.621903    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74zh6\" (UniqueName: \"kubernetes.io/projected/2323d4e8-45f0-4109-9c14-48293a0805d4-kube-api-access-74zh6\") pod \"maas-api-revoke-sub-a4910eac-f5wzv\" (UID: \"2323d4e8-45f0-4109-9c14-48293a0805d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"
Nov 02 09:54:22.708959 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:22.708879    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"
Nov 02 09:54:23.058712 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:23.058680    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"]
Nov 02 09:54:23.672020 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:23.671984    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv" event={"ID":"2323d4e8-45f0-4109-9c14-48293a0805d4","Type":"ContainerStarted","Data":"863425e043c85bf466596861be488fe952b8ac659651de0a57a097ff078e7a8f"}
Nov 02 09:54:23.672420 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:23.672046    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv" event={"ID":"2323d4e8-45f0-4109-9c14-48293a0805d4","Type":"ContainerStarted","Data":"30e9f983e8abbb571199341e5a608e870b9d6c6df5f6f6d05981b7f5ab39241e"}
Nov 02 09:54:23.693187 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:23.693075    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv" podStartSLOduration=1.693060349 podStartE2EDuration="1.693060349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:23.6901549 +0000 UTC m=+1144.522812982" watchObservedRunningTime="2026-11-02 09:54:23.693060349 +0000 UTC m=+1144.525718429"
Nov 02 09:54:24.680515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:24.680400    4614 generic.go:356] "Generic (PLEG): container finished" podID="2323d4e8-45f0-4109-9c14-48293a0805d4" containerID="863425e043c85bf466596861be488fe952b8ac659651de0a57a097ff078e7a8f" exitCode=0
Nov 02 09:54:24.680946 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:24.680483    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv" event={"ID":"2323d4e8-45f0-4109-9c14-48293a0805d4","Type":"ContainerDied","Data":"863425e043c85bf466596861be488fe952b8ac659651de0a57a097ff078e7a8f"}
Nov 02 09:54:25.403384 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:25.402731    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:25.481329 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:25.481299    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:25.742486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:25.742262    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:25.948055 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:25.948027    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"
Nov 02 09:54:26.003847 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:26.003806    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2323d4e8-45f0-4109-9c14-48293a0805d4-kube-api-access-74zh6\" (UniqueName: \"kubernetes.io/projected/2323d4e8-45f0-4109-9c14-48293a0805d4-kube-api-access-74zh6\") pod \"2323d4e8-45f0-4109-9c14-48293a0805d4\" (UID: \"2323d4e8-45f0-4109-9c14-48293a0805d4\") "
Nov 02 09:54:26.004018 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:26.003952    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2323d4e8-45f0-4109-9c14-48293a0805d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2323d4e8-45f0-4109-9c14-48293a0805d4-maas-api-openshift-ca\") pod \"2323d4e8-45f0-4109-9c14-48293a0805d4\" (UID: \"2323d4e8-45f0-4109-9c14-48293a0805d4\") "
Nov 02 09:54:26.004018 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:26.003992    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2323d4e8-45f0-4109-9c14-48293a0805d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2323d4e8-45f0-4109-9c14-48293a0805d4-maas-api-xks-ca\") pod \"2323d4e8-45f0-4109-9c14-48293a0805d4\" (UID: \"2323d4e8-45f0-4109-9c14-48293a0805d4\") "
Nov 02 09:54:26.004588 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:26.004554    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2323d4e8-45f0-4109-9c14-48293a0805d4-maas-api-openshift-ca" pod "2323d4e8-45f0-4109-9c14-48293a0805d4" (UID: "2323d4e8-45f0-4109-9c14-48293a0805d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:54:26.007048 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:26.007018    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2323d4e8-45f0-4109-9c14-48293a0805d4-kube-api-access-74zh6" pod "2323d4e8-45f0-4109-9c14-48293a0805d4" (UID: "2323d4e8-45f0-4109-9c14-48293a0805d4"). InnerVolumeSpecName "kube-api-access-74zh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:26.008760 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:26.008734    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2323d4e8-45f0-4109-9c14-48293a0805d4-maas-api-xks-ca" pod "2323d4e8-45f0-4109-9c14-48293a0805d4" (UID: "2323d4e8-45f0-4109-9c14-48293a0805d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:54:26.105697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:26.105653    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74zh6\" (UniqueName: \"kubernetes.io/projected/2323d4e8-45f0-4109-9c14-48293a0805d4-kube-api-access-74zh6\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:26.105697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:26.105695    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2323d4e8-45f0-4109-9c14-48293a0805d4-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:26.105905 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:26.105713    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2323d4e8-45f0-4109-9c14-48293a0805d4-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:26.725575 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:26.723244    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"
Nov 02 09:54:26.725575 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:26.723398    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv" event={"ID":"2323d4e8-45f0-4109-9c14-48293a0805d4","Type":"ContainerDied","Data":"30e9f983e8abbb571199341e5a608e870b9d6c6df5f6f6d05981b7f5ab39241e"}
Nov 02 09:54:26.725575 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:26.723437    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30e9f983e8abbb571199341e5a608e870b9d6c6df5f6f6d05981b7f5ab39241e"
Nov 02 09:54:28.525098 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:28.524489    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:54:29.750211 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.750166    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7363-a732ac"]
Nov 02 09:54:29.750816 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.750800    4614 cpu_manager.go:414] "containerMap: removing container" podUID="2323d4e8-45f0-4109-9c14-48293a0805d4" containerName="revoke-keys"
Nov 02 09:54:29.750866 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.750817    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2323d4e8-45f0-4109-9c14-48293a0805d4" containerName="revoke-keys"
Nov 02 09:54:29.750929 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.750919    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="2323d4e8-45f0-4109-9c14-48293a0805d4" containerName="revoke-keys"
Nov 02 09:54:29.754392 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.754374    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7363-a732ac"
Nov 02 09:54:29.759797 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.759768    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7363-a732ac"]
Nov 02 09:54:29.761220 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.761083    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x8bqk\""
Nov 02 09:54:29.863816 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.863554    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4v4t\" (UniqueName: \"kubernetes.io/projected/b5ad6885-7b26-465a-8f85-9d9640115499-kube-api-access-n4v4t\") pod \"test-curl-7363-a732ac\" (UID: \"b5ad6885-7b26-465a-8f85-9d9640115499\") " pod="opendatahub/test-curl-7363-a732ac"
Nov 02 09:54:29.876874 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.876840    4614 generic.go:356] "Generic (PLEG): container finished" podID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerID="2b3696c84c9a9a201482b7ed6b58cf3e482b26e6c47ee6fc8e0d40cdd047e963" exitCode=0
Nov 02 09:54:29.877093 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.877073    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" event={"ID":"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3","Type":"ContainerDied","Data":"2b3696c84c9a9a201482b7ed6b58cf3e482b26e6c47ee6fc8e0d40cdd047e963"}
Nov 02 09:54:29.877596 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.877572    4614 scope.go:122] "RemoveContainer" containerID="2b3696c84c9a9a201482b7ed6b58cf3e482b26e6c47ee6fc8e0d40cdd047e963"
Nov 02 09:54:29.879300 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.879266    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:29.879438 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.879379    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:29.879438 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.879429    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:29.965050 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.965017    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4v4t\" (UniqueName: \"kubernetes.io/projected/b5ad6885-7b26-465a-8f85-9d9640115499-kube-api-access-n4v4t\") pod \"test-curl-7363-a732ac\" (UID: \"b5ad6885-7b26-465a-8f85-9d9640115499\") " pod="opendatahub/test-curl-7363-a732ac"
Nov 02 09:54:29.976932 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:29.976903    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4v4t\" (UniqueName: \"kubernetes.io/projected/b5ad6885-7b26-465a-8f85-9d9640115499-kube-api-access-n4v4t\") pod \"test-curl-7363-a732ac\" (UID: \"b5ad6885-7b26-465a-8f85-9d9640115499\") " pod="opendatahub/test-curl-7363-a732ac"
Nov 02 09:54:30.031671 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:30.031588    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"]
Nov 02 09:54:30.038530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:30.038503    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfeb2c2-pjxfq"]
Nov 02 09:54:30.070175 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:30.070149    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7363-a732ac"
Nov 02 09:54:30.390639 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:30.387413    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7363-a732ac"]
Nov 02 09:54:30.391838 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:30.391807    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"]
Nov 02 09:54:30.925318 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:30.925279    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" event={"ID":"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3","Type":"ContainerStarted","Data":"b6dd4209320cd88fc333ac0b19236f5298d9b93ebef27ad4122ab0bcfd64bde6"}
Nov 02 09:54:30.957602 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:30.957565    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7363-a732ac" event={"ID":"b5ad6885-7b26-465a-8f85-9d9640115499","Type":"ContainerStarted","Data":"362143d7950e3347f75e2a5f40a3ef74aef33565ee657a80acc0b18a289c0d6c"}
Nov 02 09:54:30.957877 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:30.957852    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7363-a732ac" event={"ID":"b5ad6885-7b26-465a-8f85-9d9640115499","Type":"ContainerStarted","Data":"0ce2ba1fcbe7b7d1f8eb4c92d2d1cb13b55e552b9ecbcb594691c87a082fe9ae"}
Nov 02 09:54:30.991969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:30.991916    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7363-a732ac" podStartSLOduration=1.991902188 podStartE2EDuration="1.991902188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:30.990093926 +0000 UTC m=+1151.822752006" watchObservedRunningTime="2026-11-02 09:54:30.991902188 +0000 UTC m=+1151.824560268"
Nov 02 09:54:31.025045 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:31.025014    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:54:31.040740 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:31.029976    4614 patch_prober.go:29] interesting pod/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 09:54:31.040740 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:31.030031    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 09:54:31.316619 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:31.316587    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae0311b3-9a35-49a4-81db-2cccfc296cf2" path="/var/lib/kubelet/pods/ae0311b3-9a35-49a4-81db-2cccfc296cf2/volumes"
Nov 02 09:54:31.763377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:31.756447    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7363-a732ac"]
Nov 02 09:54:31.763377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:31.758261    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7363-a732ac"]
Nov 02 09:54:32.025163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:32.025089    4614 patch_prober.go:29] interesting pod/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 09:54:32.025163 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:32.025145    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 09:54:32.970535 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:32.970494    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7363-a732ac" podUID="b5ad6885-7b26-465a-8f85-9d9640115499" containerName="test-curl-7363-a732ac" containerID="cri-o://362143d7950e3347f75e2a5f40a3ef74aef33565ee657a80acc0b18a289c0d6c" gracePeriod=2
Nov 02 09:54:32.973066 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:54:32.973034    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7363-a732ac\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="b5ad6885-7b26-465a-8f85-9d9640115499" pod="opendatahub/test-curl-7363-a732ac"
Nov 02 09:54:33.050640 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.050602    4614 patch_prober.go:29] interesting pod/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 09:54:33.051099 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.050654    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 09:54:33.620452 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.620414    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"]
Nov 02 09:54:33.621339 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.621315    4614 cpu_manager.go:414] "containerMap: removing container" podUID="b5ad6885-7b26-465a-8f85-9d9640115499" containerName="test-curl-7363-a732ac"
Nov 02 09:54:33.621339 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.621335    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5ad6885-7b26-465a-8f85-9d9640115499" containerName="test-curl-7363-a732ac"
Nov 02 09:54:33.621651 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.621631    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5ad6885-7b26-465a-8f85-9d9640115499" containerName="test-curl-7363-a732ac"
Nov 02 09:54:33.678724 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.678688    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"]
Nov 02 09:54:33.678902 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.678804    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.762444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.762408    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.762444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.762445    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.762641 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.762539    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.762641 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.762599    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3a1400-7110-4d96-884a-f3874bf496ce-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.762641 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.762627    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.762746 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.762723    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.762796 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.762778    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.762868 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.762855    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.762921 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.762876    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jccn\" (UniqueName: \"kubernetes.io/projected/6f3a1400-7110-4d96-884a-f3874bf496ce-kube-api-access-5jccn\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.864475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.864440    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.864475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.864481    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.864732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.864534    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.864732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.864604    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3a1400-7110-4d96-884a-f3874bf496ce-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.864732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.864647    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.864899 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.864731    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.865026 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.865003    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.865091 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.865004    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.865091 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.865070    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.865200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.865120    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.865200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.865154    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jccn\" (UniqueName: \"kubernetes.io/projected/6f3a1400-7110-4d96-884a-f3874bf496ce-kube-api-access-5jccn\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.865476 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.865454    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.865577 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.865494    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.865577 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.865526    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3a1400-7110-4d96-884a-f3874bf496ce-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.867390 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.867366    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.867554 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.867535    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.877690 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.877632    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.877790 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.877776    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jccn\" (UniqueName: \"kubernetes.io/projected/6f3a1400-7110-4d96-884a-f3874bf496ce-kube-api-access-5jccn\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:33.999141 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:33.999101    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:34.049387 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:34.045101    4614 patch_prober.go:29] interesting pod/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 09:54:34.049387 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:34.045176    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 09:54:34.359955 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:54:34.359899    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3a1400_7110_4d96_884a_f3874bf496ce.slice/crio-eec9bd5255ad33a4e79e8a4f80965cdb18b0a3f35dd83b5213be134f85102348 WatchSource:0}: Error finding container eec9bd5255ad33a4e79e8a4f80965cdb18b0a3f35dd83b5213be134f85102348: Status 404 returned error can't find the container with id eec9bd5255ad33a4e79e8a4f80965cdb18b0a3f35dd83b5213be134f85102348
Nov 02 09:54:34.408323 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:34.407580    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:34.408323 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:34.407666    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:34.408323 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:34.407712    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:34.415939 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:34.415911    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"]
Nov 02 09:54:34.993598 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:34.992531    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q" event={"ID":"6f3a1400-7110-4d96-884a-f3874bf496ce","Type":"ContainerStarted","Data":"89ae424a5385219343666279fae08b0378832384d900464abc65d11ff8e29f85"}
Nov 02 09:54:34.993598 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:34.992597    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q" event={"ID":"6f3a1400-7110-4d96-884a-f3874bf496ce","Type":"ContainerStarted","Data":"eec9bd5255ad33a4e79e8a4f80965cdb18b0a3f35dd83b5213be134f85102348"}
Nov 02 09:54:34.999181 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:34.999155    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:35.025582 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:35.025019    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q" podStartSLOduration=2.024998604 podStartE2EDuration="2.024998604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:35.021721889 +0000 UTC m=+1155.854379971" watchObservedRunningTime="2026-11-02 09:54:35.024998604 +0000 UTC m=+1155.857656690"
Nov 02 09:54:35.025582 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:35.025088    4614 patch_prober.go:29] interesting pod/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 09:54:35.025582 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:35.025135    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 09:54:35.064692 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:35.064635    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"]
Nov 02 09:54:35.068744 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:35.068720    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17214e47-zpzt8"]
Nov 02 09:54:35.321346 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:35.321314    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dff8b74-45bc-40e3-8b16-4a8010a6e162" path="/var/lib/kubelet/pods/0dff8b74-45bc-40e3-8b16-4a8010a6e162/volumes"
Nov 02 09:54:35.527342 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:35.527178    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7363-a732ac"
Nov 02 09:54:35.645438 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:35.635985    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5ad6885-7b26-465a-8f85-9d9640115499-kube-api-access-n4v4t\" (UniqueName: \"kubernetes.io/projected/b5ad6885-7b26-465a-8f85-9d9640115499-kube-api-access-n4v4t\") pod \"b5ad6885-7b26-465a-8f85-9d9640115499\" (UID: \"b5ad6885-7b26-465a-8f85-9d9640115499\") "
Nov 02 09:54:35.651304 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:35.651247    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ad6885-7b26-465a-8f85-9d9640115499-kube-api-access-n4v4t" pod "b5ad6885-7b26-465a-8f85-9d9640115499" (UID: "b5ad6885-7b26-465a-8f85-9d9640115499"). InnerVolumeSpecName "kube-api-access-n4v4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:35.741375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:35.740763    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4v4t\" (UniqueName: \"kubernetes.io/projected/b5ad6885-7b26-465a-8f85-9d9640115499-kube-api-access-n4v4t\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:36.000592 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:36.000553    4614 generic.go:356] "Generic (PLEG): container finished" podID="b5ad6885-7b26-465a-8f85-9d9640115499" containerID="362143d7950e3347f75e2a5f40a3ef74aef33565ee657a80acc0b18a289c0d6c" exitCode=137
Nov 02 09:54:36.000780 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:36.000614    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7363-a732ac"
Nov 02 09:54:36.000780 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:36.000655    4614 scope.go:122] "RemoveContainer" containerID="362143d7950e3347f75e2a5f40a3ef74aef33565ee657a80acc0b18a289c0d6c"
Nov 02 09:54:36.001577 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:36.001553    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 09:54:36.001708 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:36.001602    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q" podUID="6f3a1400-7110-4d96-884a-f3874bf496ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 09:54:36.012165 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:54:36.011849    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7363-a732ac\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="b5ad6885-7b26-465a-8f85-9d9640115499" pod="opendatahub/test-curl-7363-a732ac"
Nov 02 09:54:36.012165 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:36.011989    4614 scope.go:122] "RemoveContainer" containerID="362143d7950e3347f75e2a5f40a3ef74aef33565ee657a80acc0b18a289c0d6c"
Nov 02 09:54:36.012272 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:54:36.012231    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362143d7950e3347f75e2a5f40a3ef74aef33565ee657a80acc0b18a289c0d6c\": container with ID starting with 362143d7950e3347f75e2a5f40a3ef74aef33565ee657a80acc0b18a289c0d6c not found: ID does not exist" containerID="362143d7950e3347f75e2a5f40a3ef74aef33565ee657a80acc0b18a289c0d6c"
Nov 02 09:54:36.012272 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:36.012254    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362143d7950e3347f75e2a5f40a3ef74aef33565ee657a80acc0b18a289c0d6c"} err="failed to get container status \"362143d7950e3347f75e2a5f40a3ef74aef33565ee657a80acc0b18a289c0d6c\": rpc error: code = NotFound desc = could not find container \"362143d7950e3347f75e2a5f40a3ef74aef33565ee657a80acc0b18a289c0d6c\": container with ID starting with 362143d7950e3347f75e2a5f40a3ef74aef33565ee657a80acc0b18a289c0d6c not found: ID does not exist"
Nov 02 09:54:36.047377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:36.042767    4614 patch_prober.go:29] interesting pod/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 09:54:36.047377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:36.042830    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 09:54:36.668078 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:36.668044    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:54:37.011576 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:37.011545    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 09:54:37.011844 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:37.011816    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q" podUID="6f3a1400-7110-4d96-884a-f3874bf496ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 09:54:37.060781 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:37.060740    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:54:37.072708 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:37.072671    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:54:37.072987 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:54:37.072682    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7363-a732ac\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="b5ad6885-7b26-465a-8f85-9d9640115499" pod="opendatahub/test-curl-7363-a732ac"
Nov 02 09:54:37.103737 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:54:37.103688    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7363-a732ac\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="b5ad6885-7b26-465a-8f85-9d9640115499" pod="opendatahub/test-curl-7363-a732ac"
Nov 02 09:54:37.575063 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:37.575024    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5ad6885-7b26-465a-8f85-9d9640115499" path="/var/lib/kubelet/pods/b5ad6885-7b26-465a-8f85-9d9640115499/volumes"
Nov 02 09:54:38.014375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:38.011434    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 09:54:38.014375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:38.011512    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q" podUID="6f3a1400-7110-4d96-884a-f3874bf496ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 09:54:39.000021 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:38.999985    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 09:54:39.000336 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.000307    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q" podUID="6f3a1400-7110-4d96-884a-f3874bf496ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 09:54:39.080225 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.080106    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"]
Nov 02 09:54:39.084821 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.084798    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.090535 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.090514    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a71884-openshift-default-dockercfg-lthtw\""
Nov 02 09:54:39.103278 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.103242    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"]
Nov 02 09:54:39.218379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.216229    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-workload-certs\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.218379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.216396    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-podinfo\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.218379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.216450    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-token\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.218379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.216488    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xxmb\" (UniqueName: \"kubernetes.io/projected/bac35651-e552-4e21-89c9-c76ee6d72ec4-kube-api-access-5xxmb\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.218379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.216573    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-credential-socket\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.218379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.216619    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-workload-socket\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.218379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.216644    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac35651-e552-4e21-89c9-c76ee6d72ec4-istiod-ca-cert\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.218379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.216730    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-envoy\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.218379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.216753    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-data\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.318220    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-envoy\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.318314    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-data\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.318405    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-workload-certs\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.318523    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-podinfo\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.318575    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-token\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.318613    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xxmb\" (UniqueName: \"kubernetes.io/projected/bac35651-e552-4e21-89c9-c76ee6d72ec4-kube-api-access-5xxmb\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.318699    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-credential-socket\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.318743    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-workload-socket\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.318767    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac35651-e552-4e21-89c9-c76ee6d72ec4-istiod-ca-cert\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.320001    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac35651-e552-4e21-89c9-c76ee6d72ec4-istiod-ca-cert\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.321169    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-data\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.321493    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-workload-certs\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.321774    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-credential-socket\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.322232    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-workload-socket\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.335444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.335153    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-token\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.343683 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.343623    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-podinfo\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.364038 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.363962    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-envoy\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.373386 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.367068    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xxmb\" (UniqueName: \"kubernetes.io/projected/bac35651-e552-4e21-89c9-c76ee6d72ec4-kube-api-access-5xxmb\") pod \"e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") " pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.419888 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.419612    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:39.694906 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:54:39.694796    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:54:39.974376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.971381    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:39.974376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.972026    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:39.974376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.972303    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:54:39.976142 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.976112    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"]
Nov 02 09:54:39.979627 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:54:39.979573    4614 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac35651_e552_4e21_89c9_c76ee6d72ec4.slice/crio-309476ebc790e44f4e160b1aad662fd3c6936f4ab663f682d9df5c033a87d113\": RecentStats: unable to find data in memory cache]"
Nov 02 09:54:39.999742 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.999716    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 09:54:39.999909 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:39.999882    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q" podUID="6f3a1400-7110-4d96-884a-f3874bf496ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 09:54:40.090567 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:40.090399    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"]
Nov 02 09:54:40.098683 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:40.098646    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-950b3b12-k8rwn"]
Nov 02 09:54:40.121620 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:40.120143    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" event={"ID":"bac35651-e552-4e21-89c9-c76ee6d72ec4","Type":"ContainerStarted","Data":"309476ebc790e44f4e160b1aad662fd3c6936f4ab663f682d9df5c033a87d113"}
Nov 02 09:54:41.003738 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:41.003700    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 09:54:41.003912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:41.003759    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q" podUID="6f3a1400-7110-4d96-884a-f3874bf496ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 09:54:41.234376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:41.219556    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" event={"ID":"bac35651-e552-4e21-89c9-c76ee6d72ec4","Type":"ContainerStarted","Data":"144bb1ac9d3051e67ca9e1d5e015a16ed4e0e120402ffe6a8892b42c84fdf5ea"}
Nov 02 09:54:41.268041 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:41.267916    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" podStartSLOduration=2.267895301 podStartE2EDuration="2.267895301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:54:41.262834443 +0000 UTC m=+1162.095492524" watchObservedRunningTime="2026-11-02 09:54:41.267895301 +0000 UTC m=+1162.100553386"
Nov 02 09:54:41.347466 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:41.347429    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d67cd3d-26dd-4e86-9445-f011a3be1666" path="/var/lib/kubelet/pods/1d67cd3d-26dd-4e86-9445-f011a3be1666/volumes"
Nov 02 09:54:41.423951 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:41.423725    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:41.446124 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:41.445809    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:42.241377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:42.241305    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:42.243748 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:42.243345    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:54:42.424590 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:42.424559    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:42.424902 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:42.424883    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:42.425724 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:42.425705    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 09:54:45.426971 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.426912    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t" podUID="6ad89d95-c57a-47e7-ac5b-efca4018cfbd" containerName="main" containerID="cri-o://841d4201443920aedfc5b51be349d6e48372d83a5828247bfa9751ca75b01750" gracePeriod=45
Nov 02 09:54:45.710307 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.710225    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t" podUID="6ad89d95-c57a-47e7-ac5b-efca4018cfbd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": dial tcp 10.132.0.110:8000: connect: connection refused"
Nov 02 09:54:45.840763 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.840741    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:45.905530 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.905495    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-model-cache\") pod \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") "
Nov 02 09:54:45.905726 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.905670    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-kube-api-access-whwq7\" (UniqueName: \"kubernetes.io/projected/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-kube-api-access-whwq7\") pod \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") "
Nov 02 09:54:45.905798 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.905762    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-dshm\") pod \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") "
Nov 02 09:54:45.905860 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.905797    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-tls-certs\") pod \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") "
Nov 02 09:54:45.905860 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.905839    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-model-cache" pod "6ad89d95-c57a-47e7-ac5b-efca4018cfbd" (UID: "6ad89d95-c57a-47e7-ac5b-efca4018cfbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:54:45.905956 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.905929    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-tmp-dir\") pod \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") "
Nov 02 09:54:45.906008 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.905955    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-home\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-home\") pod \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\" (UID: \"6ad89d95-c57a-47e7-ac5b-efca4018cfbd\") "
Nov 02 09:54:45.906454 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.906336    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-tmp-dir" pod "6ad89d95-c57a-47e7-ac5b-efca4018cfbd" (UID: "6ad89d95-c57a-47e7-ac5b-efca4018cfbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:54:45.906454 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.906422    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-home" pod "6ad89d95-c57a-47e7-ac5b-efca4018cfbd" (UID: "6ad89d95-c57a-47e7-ac5b-efca4018cfbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:54:45.906640 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.906620    4614 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-tmp-dir\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:45.906688 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.906647    4614 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-home\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:45.906688 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.906661    4614 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-model-cache\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:45.909001 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.908969    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-dshm" pod "6ad89d95-c57a-47e7-ac5b-efca4018cfbd" (UID: "6ad89d95-c57a-47e7-ac5b-efca4018cfbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:54:45.909001 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.908983    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-tls-certs" pod "6ad89d95-c57a-47e7-ac5b-efca4018cfbd" (UID: "6ad89d95-c57a-47e7-ac5b-efca4018cfbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:54:45.909001 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:45.908976    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-kube-api-access-whwq7" pod "6ad89d95-c57a-47e7-ac5b-efca4018cfbd" (UID: "6ad89d95-c57a-47e7-ac5b-efca4018cfbd"). InnerVolumeSpecName "kube-api-access-whwq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:54:46.007632 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:46.007604    4614 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-dshm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:46.007632 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:46.007631    4614 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-tls-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:46.007835 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:46.007641    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whwq7\" (UniqueName: \"kubernetes.io/projected/6ad89d95-c57a-47e7-ac5b-efca4018cfbd-kube-api-access-whwq7\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:54:46.333636 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:46.333541    4614 generic.go:356] "Generic (PLEG): container finished" podID="6ad89d95-c57a-47e7-ac5b-efca4018cfbd" containerID="841d4201443920aedfc5b51be349d6e48372d83a5828247bfa9751ca75b01750" exitCode=0
Nov 02 09:54:46.333636 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:46.333589    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"
Nov 02 09:54:46.333636 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:46.333622    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t" event={"ID":"6ad89d95-c57a-47e7-ac5b-efca4018cfbd","Type":"ContainerDied","Data":"841d4201443920aedfc5b51be349d6e48372d83a5828247bfa9751ca75b01750"}
Nov 02 09:54:46.333898 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:46.333655    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t" event={"ID":"6ad89d95-c57a-47e7-ac5b-efca4018cfbd","Type":"ContainerDied","Data":"4119e3b488879c4a56fd404f100efe57d398d2100c916eab61d70041259b3610"}
Nov 02 09:54:46.333898 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:46.333676    4614 scope.go:122] "RemoveContainer" containerID="841d4201443920aedfc5b51be349d6e48372d83a5828247bfa9751ca75b01750"
Nov 02 09:54:46.344550 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:46.344503    4614 scope.go:122] "RemoveContainer" containerID="841d4201443920aedfc5b51be349d6e48372d83a5828247bfa9751ca75b01750"
Nov 02 09:54:46.344798 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:54:46.344777    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841d4201443920aedfc5b51be349d6e48372d83a5828247bfa9751ca75b01750\": container with ID starting with 841d4201443920aedfc5b51be349d6e48372d83a5828247bfa9751ca75b01750 not found: ID does not exist" containerID="841d4201443920aedfc5b51be349d6e48372d83a5828247bfa9751ca75b01750"
Nov 02 09:54:46.344869 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:46.344811    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841d4201443920aedfc5b51be349d6e48372d83a5828247bfa9751ca75b01750"} err="failed to get container status \"841d4201443920aedfc5b51be349d6e48372d83a5828247bfa9751ca75b01750\": rpc error: code = NotFound desc = could not find container \"841d4201443920aedfc5b51be349d6e48372d83a5828247bfa9751ca75b01750\": container with ID starting with 841d4201443920aedfc5b51be349d6e48372d83a5828247bfa9751ca75b01750 not found: ID does not exist"
Nov 02 09:54:46.357838 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:46.357815    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"]
Nov 02 09:54:46.363491 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:46.363471    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-ced1b7/autores-model-ced1b7-kserve-685cc8796f-c7k6t"]
Nov 02 09:54:47.323245 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:54:47.323203    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad89d95-c57a-47e7-ac5b-efca4018cfbd" path="/var/lib/kubelet/pods/6ad89d95-c57a-47e7-ac5b-efca4018cfbd/volumes"
Nov 02 09:55:09.635618 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:09.635582    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:55:09.929045 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:09.928953    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"]
Nov 02 09:55:09.929625 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:09.929590    4614 cpu_manager.go:414] "containerMap: removing container" podUID="6ad89d95-c57a-47e7-ac5b-efca4018cfbd" containerName="main"
Nov 02 09:55:09.929625 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:09.929618    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad89d95-c57a-47e7-ac5b-efca4018cfbd" containerName="main"
Nov 02 09:55:09.929808 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:09.929713    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad89d95-c57a-47e7-ac5b-efca4018cfbd" containerName="main"
Nov 02 09:55:09.940397 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:09.934151    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:09.950406 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:09.944829    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ae13d4-openshift-default-dockercfg-46zl4\""
Nov 02 09:55:09.955666 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:09.955637    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"]
Nov 02 09:55:10.086369 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.086308    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-workload-socket\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.086687 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.086662    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-workload-certs\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.086846 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.086827    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cqtt\" (UniqueName: \"kubernetes.io/projected/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-kube-api-access-7cqtt\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.087138 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.087121    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-credential-socket\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.087267 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.087252    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istiod-ca-cert\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.087416 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.087400    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-data\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.087588 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.087573    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-token\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.087708 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.087694    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-podinfo\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.087806 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.087793    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-envoy\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.208463 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.201243    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-credential-socket\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.208463 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.201320    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istiod-ca-cert\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.208463 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.201390    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-data\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.208463 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.201464    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-token\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.208463 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.201498    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-podinfo\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.208463 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.201522    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-envoy\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.208463 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.201584    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-workload-socket\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.208463 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.201618    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-workload-certs\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.208463 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.201656    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cqtt\" (UniqueName: \"kubernetes.io/projected/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-kube-api-access-7cqtt\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.208463 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.202618    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-credential-socket\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.208463 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.203246    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istiod-ca-cert\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.209924 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.203516    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-data\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.210730 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.210413    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-workload-certs\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.211136 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.211116    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-workload-socket\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.294377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.277120    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-envoy\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.294956 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.294888    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-podinfo\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.319801 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.318542    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cqtt\" (UniqueName: \"kubernetes.io/projected/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-kube-api-access-7cqtt\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.345932 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.345866    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-token\") pod \"e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") " pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.586607 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.586367    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:10.927325 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.927235    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"]
Nov 02 09:55:10.937021 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:55:10.936987    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5973ac7_7b39_4db6_bb4b_56014b7bf3b2.slice/crio-29cfce08e4f3ea99175a77051bbba7129808038a909c70573e109f2fbefae979 WatchSource:0}: Error finding container 29cfce08e4f3ea99175a77051bbba7129808038a909c70573e109f2fbefae979: Status 404 returned error can't find the container with id 29cfce08e4f3ea99175a77051bbba7129808038a909c70573e109f2fbefae979
Nov 02 09:55:10.940063 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.940021    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:10.940184 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.940111    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:10.940184 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:10.940152    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:11.593644 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:11.593596    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" event={"ID":"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2","Type":"ContainerStarted","Data":"d67c97639598c40189817f3e3d6897e3450c7f73ae1b62758a96115850cd2fad"}
Nov 02 09:55:11.593885 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:11.593695    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" event={"ID":"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2","Type":"ContainerStarted","Data":"29cfce08e4f3ea99175a77051bbba7129808038a909c70573e109f2fbefae979"}
Nov 02 09:55:11.622513 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:11.622457    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" podStartSLOduration=2.622443111 podStartE2EDuration="2.622443111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:11.621895702 +0000 UTC m=+1192.454553782" watchObservedRunningTime="2026-11-02 09:55:11.622443111 +0000 UTC m=+1192.455101192"
Nov 02 09:55:12.586682 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:12.586649    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:12.592656 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:12.592634    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:12.600601 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:12.600577    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:12.601313 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:12.601292    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:12.664399 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:12.660962    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"]
Nov 02 09:55:13.283532 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:13.283500    4614 patch_prober.go:29] interesting pod/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:55:13.283710 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:13.283555    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:13.657436 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:13.655505    4614 generic.go:356] "Generic (PLEG): container finished" podID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerID="144bb1ac9d3051e67ca9e1d5e015a16ed4e0e120402ffe6a8892b42c84fdf5ea" exitCode=0
Nov 02 09:55:13.657436 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:13.656743    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" event={"ID":"bac35651-e552-4e21-89c9-c76ee6d72ec4","Type":"ContainerDied","Data":"144bb1ac9d3051e67ca9e1d5e015a16ed4e0e120402ffe6a8892b42c84fdf5ea"}
Nov 02 09:55:13.657436 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:13.657185    4614 scope.go:122] "RemoveContainer" containerID="144bb1ac9d3051e67ca9e1d5e015a16ed4e0e120402ffe6a8892b42c84fdf5ea"
Nov 02 09:55:13.659162 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:13.659076    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:13.659841 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:13.659817    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:13.660015 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:13.659997    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:14.677448 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:14.677414    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" event={"ID":"bac35651-e552-4e21-89c9-c76ee6d72ec4","Type":"ContainerStarted","Data":"bc0c235faea8ffbaaef6f6b3f9b140001b1e19f6db600f446f52bdde6ae4f061"}
Nov 02 09:55:15.451375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:15.446549    4614 patch_prober.go:29] interesting pod/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 09:55:15.451375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:15.446742    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 09:55:15.499614 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:15.499560    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:55:16.445437 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:16.445403    4614 patch_prober.go:29] interesting pod/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 09:55:16.446650 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:16.446615    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 09:55:17.420665 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:17.420624    4614 patch_prober.go:29] interesting pod/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 09:55:17.420878 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:17.420705    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 09:55:17.663778 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:17.663732    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8" podUID="0507afd5-3cdb-4432-8c5e-3322dddd7ed9" containerName="payload-pre-processing" containerID="cri-o://507cd33d897c3499a3078acb6d2bd7a1ef02ce8c96cffab573fa1b6a2251e465" gracePeriod=25
Nov 02 09:55:18.318593 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.318560    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"
Nov 02 09:55:18.422200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.422163    4614 patch_prober.go:29] interesting pod/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 09:55:18.422426 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.422228    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 09:55:18.463379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.459934    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0507afd5-3cdb-4432-8c5e-3322dddd7ed9-kube-api-access-chh2k\" (UniqueName: \"kubernetes.io/projected/0507afd5-3cdb-4432-8c5e-3322dddd7ed9-kube-api-access-chh2k\") pod \"0507afd5-3cdb-4432-8c5e-3322dddd7ed9\" (UID: \"0507afd5-3cdb-4432-8c5e-3322dddd7ed9\") "
Nov 02 09:55:18.463379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.460081    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0507afd5-3cdb-4432-8c5e-3322dddd7ed9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0507afd5-3cdb-4432-8c5e-3322dddd7ed9-plugins-config-volume\") pod \"0507afd5-3cdb-4432-8c5e-3322dddd7ed9\" (UID: \"0507afd5-3cdb-4432-8c5e-3322dddd7ed9\") "
Nov 02 09:55:18.463379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.461394    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0507afd5-3cdb-4432-8c5e-3322dddd7ed9-plugins-config-volume" pod "0507afd5-3cdb-4432-8c5e-3322dddd7ed9" (UID: "0507afd5-3cdb-4432-8c5e-3322dddd7ed9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:18.469402 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.464579    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0507afd5-3cdb-4432-8c5e-3322dddd7ed9-kube-api-access-chh2k" pod "0507afd5-3cdb-4432-8c5e-3322dddd7ed9" (UID: "0507afd5-3cdb-4432-8c5e-3322dddd7ed9"). InnerVolumeSpecName "kube-api-access-chh2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:18.474384 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.471722    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chh2k\" (UniqueName: \"kubernetes.io/projected/0507afd5-3cdb-4432-8c5e-3322dddd7ed9-kube-api-access-chh2k\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:18.474384 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.471765    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0507afd5-3cdb-4432-8c5e-3322dddd7ed9-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:18.810798 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.810760    4614 generic.go:356] "Generic (PLEG): container finished" podID="0507afd5-3cdb-4432-8c5e-3322dddd7ed9" containerID="507cd33d897c3499a3078acb6d2bd7a1ef02ce8c96cffab573fa1b6a2251e465" exitCode=0
Nov 02 09:55:18.811324 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.810832    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"
Nov 02 09:55:18.811532 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.810851    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8" event={"ID":"0507afd5-3cdb-4432-8c5e-3322dddd7ed9","Type":"ContainerDied","Data":"507cd33d897c3499a3078acb6d2bd7a1ef02ce8c96cffab573fa1b6a2251e465"}
Nov 02 09:55:18.811779 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.811584    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8" event={"ID":"0507afd5-3cdb-4432-8c5e-3322dddd7ed9","Type":"ContainerDied","Data":"50d316bdc75eebd62b8eb5d477af01e7a2d5618ca4fd2457d163619713f12934"}
Nov 02 09:55:18.811779 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.811626    4614 scope.go:122] "RemoveContainer" containerID="507cd33d897c3499a3078acb6d2bd7a1ef02ce8c96cffab573fa1b6a2251e465"
Nov 02 09:55:18.838576 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.835705    4614 scope.go:122] "RemoveContainer" containerID="507cd33d897c3499a3078acb6d2bd7a1ef02ce8c96cffab573fa1b6a2251e465"
Nov 02 09:55:18.849384 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:18.846509    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507cd33d897c3499a3078acb6d2bd7a1ef02ce8c96cffab573fa1b6a2251e465\": container with ID starting with 507cd33d897c3499a3078acb6d2bd7a1ef02ce8c96cffab573fa1b6a2251e465 not found: ID does not exist" containerID="507cd33d897c3499a3078acb6d2bd7a1ef02ce8c96cffab573fa1b6a2251e465"
Nov 02 09:55:18.849384 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.846551    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507cd33d897c3499a3078acb6d2bd7a1ef02ce8c96cffab573fa1b6a2251e465"} err="failed to get container status \"507cd33d897c3499a3078acb6d2bd7a1ef02ce8c96cffab573fa1b6a2251e465\": rpc error: code = NotFound desc = could not find container \"507cd33d897c3499a3078acb6d2bd7a1ef02ce8c96cffab573fa1b6a2251e465\": container with ID starting with 507cd33d897c3499a3078acb6d2bd7a1ef02ce8c96cffab573fa1b6a2251e465 not found: ID does not exist"
Nov 02 09:55:18.857542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.857512    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"]
Nov 02 09:55:18.862033 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.862012    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ced1b7-7557c8bd4c-pslr8"]
Nov 02 09:55:18.987556 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.986891    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"]
Nov 02 09:55:18.987556 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:18.987174    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn" podUID="44b72964-98f0-44cb-9b75-6e79e4c8a17d" containerName="maas-api" containerID="cri-o://d6d1600000870604bdbc12748bc2c10b2eeafa927498ba543ec10f7b93d2abee" gracePeriod=30
Nov 02 09:55:19.447094 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.445382    4614 patch_prober.go:29] interesting pod/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 09:55:19.447094 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.445455    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 09:55:19.511437 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.511393    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0507afd5-3cdb-4432-8c5e-3322dddd7ed9" path="/var/lib/kubelet/pods/0507afd5-3cdb-4432-8c5e-3322dddd7ed9/volumes"
Nov 02 09:55:19.603546 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.603515    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:55:19.707641 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.707564    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 09:55:19.708593 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.708565    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 09:55:19.714726 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.714699    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b72964-98f0-44cb-9b75-6e79e4c8a17d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b72964-98f0-44cb-9b75-6e79e4c8a17d-metrics-certs\") pod \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") "
Nov 02 09:55:19.714869 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.714760    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b72964-98f0-44cb-9b75-6e79e4c8a17d-kube-api-access-nx7cl\" (UniqueName: \"kubernetes.io/projected/44b72964-98f0-44cb-9b75-6e79e4c8a17d-kube-api-access-nx7cl\") pod \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") "
Nov 02 09:55:19.715206 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.715181    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b72964-98f0-44cb-9b75-6e79e4c8a17d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b72964-98f0-44cb-9b75-6e79e4c8a17d-maas-api-tls\") pod \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") "
Nov 02 09:55:19.715336 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.715300    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b72964-98f0-44cb-9b75-6e79e4c8a17d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b72964-98f0-44cb-9b75-6e79e4c8a17d-openshift-service-ca\") pod \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\" (UID: \"44b72964-98f0-44cb-9b75-6e79e4c8a17d\") "
Nov 02 09:55:19.715803 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.715765    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b72964-98f0-44cb-9b75-6e79e4c8a17d-openshift-service-ca" pod "44b72964-98f0-44cb-9b75-6e79e4c8a17d" (UID: "44b72964-98f0-44cb-9b75-6e79e4c8a17d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:19.716733 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.716013    4614 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b72964-98f0-44cb-9b75-6e79e4c8a17d-openshift-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:19.721700 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.721657    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b72964-98f0-44cb-9b75-6e79e4c8a17d-metrics-certs" pod "44b72964-98f0-44cb-9b75-6e79e4c8a17d" (UID: "44b72964-98f0-44cb-9b75-6e79e4c8a17d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:55:19.722171 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.722152    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b72964-98f0-44cb-9b75-6e79e4c8a17d-maas-api-tls" pod "44b72964-98f0-44cb-9b75-6e79e4c8a17d" (UID: "44b72964-98f0-44cb-9b75-6e79e4c8a17d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:55:19.756612 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.740977    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b72964-98f0-44cb-9b75-6e79e4c8a17d-kube-api-access-nx7cl" pod "44b72964-98f0-44cb-9b75-6e79e4c8a17d" (UID: "44b72964-98f0-44cb-9b75-6e79e4c8a17d"). InnerVolumeSpecName "kube-api-access-nx7cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:19.817931 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.817898    4614 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b72964-98f0-44cb-9b75-6e79e4c8a17d-metrics-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:19.817931 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.817933    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx7cl\" (UniqueName: \"kubernetes.io/projected/44b72964-98f0-44cb-9b75-6e79e4c8a17d-kube-api-access-nx7cl\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:19.818385 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.817950    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b72964-98f0-44cb-9b75-6e79e4c8a17d-maas-api-tls\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:19.819852 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.819824    4614 generic.go:356] "Generic (PLEG): container finished" podID="44b72964-98f0-44cb-9b75-6e79e4c8a17d" containerID="d6d1600000870604bdbc12748bc2c10b2eeafa927498ba543ec10f7b93d2abee" exitCode=0
Nov 02 09:55:19.819997 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.819879    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"
Nov 02 09:55:19.819997 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.819913    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn" event={"ID":"44b72964-98f0-44cb-9b75-6e79e4c8a17d","Type":"ContainerDied","Data":"d6d1600000870604bdbc12748bc2c10b2eeafa927498ba543ec10f7b93d2abee"}
Nov 02 09:55:19.819997 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.819990    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn" event={"ID":"44b72964-98f0-44cb-9b75-6e79e4c8a17d","Type":"ContainerDied","Data":"e2668755028d218a19d3fb8ef72a8d7180e0608e8a2a6408ecd5d096de91b27a"}
Nov 02 09:55:19.820165 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.820014    4614 scope.go:122] "RemoveContainer" containerID="d6d1600000870604bdbc12748bc2c10b2eeafa927498ba543ec10f7b93d2abee"
Nov 02 09:55:19.929378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.925663    4614 scope.go:122] "RemoveContainer" containerID="d6d1600000870604bdbc12748bc2c10b2eeafa927498ba543ec10f7b93d2abee"
Nov 02 09:55:19.929378 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:19.926157    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d1600000870604bdbc12748bc2c10b2eeafa927498ba543ec10f7b93d2abee\": container with ID starting with d6d1600000870604bdbc12748bc2c10b2eeafa927498ba543ec10f7b93d2abee not found: ID does not exist" containerID="d6d1600000870604bdbc12748bc2c10b2eeafa927498ba543ec10f7b93d2abee"
Nov 02 09:55:19.929378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.926196    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d1600000870604bdbc12748bc2c10b2eeafa927498ba543ec10f7b93d2abee"} err="failed to get container status \"d6d1600000870604bdbc12748bc2c10b2eeafa927498ba543ec10f7b93d2abee\": rpc error: code = NotFound desc = could not find container \"d6d1600000870604bdbc12748bc2c10b2eeafa927498ba543ec10f7b93d2abee\": container with ID starting with d6d1600000870604bdbc12748bc2c10b2eeafa927498ba543ec10f7b93d2abee not found: ID does not exist"
Nov 02 09:55:19.951375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.943489    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"]
Nov 02 09:55:19.984376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:19.980630    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn"]
Nov 02 09:55:20.130377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:20.124545    4614 scope.go:122] "RemoveContainer" containerID="ede096370a01f03fb23abd46338ca2e57f528f282f6ee3190730778bf6189eb2"
Nov 02 09:55:20.174562 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:20.171795    4614 scope.go:122] "RemoveContainer" containerID="3f41c935363e047a5fea6f875472534a6f6305abd6e0d676ff622ee4d0c07a56"
Nov 02 09:55:20.436374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:20.428010    4614 patch_prober.go:29] interesting pod/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 02 09:55:20.436374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:20.428124    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 02 09:55:21.316203 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:21.316171    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b72964-98f0-44cb-9b75-6e79e4c8a17d" path="/var/lib/kubelet/pods/44b72964-98f0-44cb-9b75-6e79e4c8a17d/volumes"
Nov 02 09:55:21.427263 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:21.427227    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"]
Nov 02 09:55:21.428710 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:21.427512    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerName="istio-proxy" containerID="cri-o://b6dd4209320cd88fc333ac0b19236f5298d9b93ebef27ad4122ab0bcfd64bde6" gracePeriod=30
Nov 02 09:55:21.428710 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:21.428401    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:55:21.429272 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:21.428858    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:55:21.439575 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:21.439555    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:55:22.065055 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:22.065017    4614 patch_prober.go:29] interesting pod/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 09:55:22.065204 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:22.065073    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 09:55:22.295051 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:22.295008    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9cbd331c-7cc9849c48-gd4gn" podUID="44b72964-98f0-44cb-9b75-6e79e4c8a17d" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.108:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:23.737174 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:23.737142    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"]
Nov 02 09:55:23.737948 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:23.737916    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j" podUID="7b48362a-5918-4899-abac-14f674804556" containerName="istio-proxy" containerID="cri-o://28e175dbd047d908170c47865844ddc2cc9aa190d5c55b0e61f3d44b6d6da72c" gracePeriod=30
Nov 02 09:55:26.717344 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.717274    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:55:26.834546 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.834513    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-token\" (UniqueName: \"kubernetes.io/projected/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-token\") pod \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") "
Nov 02 09:55:26.834726 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.834567    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-envoy\") pod \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") "
Nov 02 09:55:26.834726 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.834595    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-workload-certs\") pod \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") "
Nov 02 09:55:26.834726 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.834682    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istiod-ca-cert\") pod \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") "
Nov 02 09:55:26.834908 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.834734    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-credential-socket\") pod \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") "
Nov 02 09:55:26.834908 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.834765    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-data\") pod \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") "
Nov 02 09:55:26.834908 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.834793    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-workload-socket\") pod \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") "
Nov 02 09:55:26.834908 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.834898    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-kube-api-access-d68nq\" (UniqueName: \"kubernetes.io/projected/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-kube-api-access-d68nq\") pod \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") "
Nov 02 09:55:26.835122 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.834950    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-podinfo\") pod \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\" (UID: \"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3\") "
Nov 02 09:55:26.835191 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.835093    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-workload-socket" pod "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" (UID: "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:26.835265 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.835244    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istiod-ca-cert" pod "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" (UID: "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:26.835766 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.835740    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:26.835883 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.835770    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:26.836109 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.836080    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-data" pod "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" (UID: "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:26.836195 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.835027    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-credential-socket" pod "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" (UID: "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:26.836304 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.836283    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-workload-certs" pod "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" (UID: "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:26.837250 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.837220    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-envoy" pod "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" (UID: "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:26.837417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.837395    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-token" pod "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" (UID: "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:26.837562 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.837542    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-kube-api-access-d68nq" pod "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" (UID: "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3"). InnerVolumeSpecName "kube-api-access-d68nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:26.837618 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.837593    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-podinfo" pod "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" (UID: "7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:55:26.901727 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.901696    4614 generic.go:356] "Generic (PLEG): container finished" podID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerID="b6dd4209320cd88fc333ac0b19236f5298d9b93ebef27ad4122ab0bcfd64bde6" exitCode=0
Nov 02 09:55:26.901894 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.901757    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"
Nov 02 09:55:26.901894 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.901784    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" event={"ID":"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3","Type":"ContainerDied","Data":"b6dd4209320cd88fc333ac0b19236f5298d9b93ebef27ad4122ab0bcfd64bde6"}
Nov 02 09:55:26.901894 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.901859    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h" event={"ID":"7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3","Type":"ContainerDied","Data":"97459a5bf72aead6deb8caf02594b315c2bc9ee17fcda2c0434709e1c5f99250"}
Nov 02 09:55:26.902053 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.901893    4614 scope.go:122] "RemoveContainer" containerID="b6dd4209320cd88fc333ac0b19236f5298d9b93ebef27ad4122ab0bcfd64bde6"
Nov 02 09:55:26.914173 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.914153    4614 scope.go:122] "RemoveContainer" containerID="2b3696c84c9a9a201482b7ed6b58cf3e482b26e6c47ee6fc8e0d40cdd047e963"
Nov 02 09:55:26.925293 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.925270    4614 scope.go:122] "RemoveContainer" containerID="b6dd4209320cd88fc333ac0b19236f5298d9b93ebef27ad4122ab0bcfd64bde6"
Nov 02 09:55:26.935397 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:26.927684    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6dd4209320cd88fc333ac0b19236f5298d9b93ebef27ad4122ab0bcfd64bde6\": container with ID starting with b6dd4209320cd88fc333ac0b19236f5298d9b93ebef27ad4122ab0bcfd64bde6 not found: ID does not exist" containerID="b6dd4209320cd88fc333ac0b19236f5298d9b93ebef27ad4122ab0bcfd64bde6"
Nov 02 09:55:26.935397 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.927737    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6dd4209320cd88fc333ac0b19236f5298d9b93ebef27ad4122ab0bcfd64bde6"} err="failed to get container status \"b6dd4209320cd88fc333ac0b19236f5298d9b93ebef27ad4122ab0bcfd64bde6\": rpc error: code = NotFound desc = could not find container \"b6dd4209320cd88fc333ac0b19236f5298d9b93ebef27ad4122ab0bcfd64bde6\": container with ID starting with b6dd4209320cd88fc333ac0b19236f5298d9b93ebef27ad4122ab0bcfd64bde6 not found: ID does not exist"
Nov 02 09:55:26.935397 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.927762    4614 scope.go:122] "RemoveContainer" containerID="2b3696c84c9a9a201482b7ed6b58cf3e482b26e6c47ee6fc8e0d40cdd047e963"
Nov 02 09:55:26.935397 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:26.929866    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3696c84c9a9a201482b7ed6b58cf3e482b26e6c47ee6fc8e0d40cdd047e963\": container with ID starting with 2b3696c84c9a9a201482b7ed6b58cf3e482b26e6c47ee6fc8e0d40cdd047e963 not found: ID does not exist" containerID="2b3696c84c9a9a201482b7ed6b58cf3e482b26e6c47ee6fc8e0d40cdd047e963"
Nov 02 09:55:26.935397 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.929901    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3696c84c9a9a201482b7ed6b58cf3e482b26e6c47ee6fc8e0d40cdd047e963"} err="failed to get container status \"2b3696c84c9a9a201482b7ed6b58cf3e482b26e6c47ee6fc8e0d40cdd047e963\": rpc error: code = NotFound desc = could not find container \"2b3696c84c9a9a201482b7ed6b58cf3e482b26e6c47ee6fc8e0d40cdd047e963\": container with ID starting with 2b3696c84c9a9a201482b7ed6b58cf3e482b26e6c47ee6fc8e0d40cdd047e963 not found: ID does not exist"
Nov 02 09:55:26.937720 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.935775    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"]
Nov 02 09:55:26.937720 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.937567    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:26.937720 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.937591    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:26.937720 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.937613    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d68nq\" (UniqueName: \"kubernetes.io/projected/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-kube-api-access-d68nq\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:26.937720 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.937639    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:26.937720 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.937658    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:26.937720 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.937672    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:26.937720 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.937687    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:26.940612 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:26.939843    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-ced1b7-openshift-default-7d96c7f879-hgv8h"]
Nov 02 09:55:27.316903 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:27.316868    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" path="/var/lib/kubelet/pods/7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3/volumes"
Nov 02 09:55:27.372797 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:27.372748    4614 patch_prober.go:29] interesting pod/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 09:55:27.372984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:27.372812    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j" podUID="7b48362a-5918-4899-abac-14f674804556" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 09:55:28.379206 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.379173    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"]
Nov 02 09:55:28.379926 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.379904    4614 cpu_manager.go:414] "containerMap: removing container" podUID="0507afd5-3cdb-4432-8c5e-3322dddd7ed9" containerName="payload-pre-processing"
Nov 02 09:55:28.380023 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.379927    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0507afd5-3cdb-4432-8c5e-3322dddd7ed9" containerName="payload-pre-processing"
Nov 02 09:55:28.380023 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.379951    4614 cpu_manager.go:414] "containerMap: removing container" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerName="istio-proxy"
Nov 02 09:55:28.380023 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.379960    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerName="istio-proxy"
Nov 02 09:55:28.380023 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.379979    4614 cpu_manager.go:414] "containerMap: removing container" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerName="istio-proxy"
Nov 02 09:55:28.380023 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.379987    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerName="istio-proxy"
Nov 02 09:55:28.380023 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.379999    4614 cpu_manager.go:414] "containerMap: removing container" podUID="44b72964-98f0-44cb-9b75-6e79e4c8a17d" containerName="maas-api"
Nov 02 09:55:28.380023 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.380007    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b72964-98f0-44cb-9b75-6e79e4c8a17d" containerName="maas-api"
Nov 02 09:55:28.380386 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.380166    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="0507afd5-3cdb-4432-8c5e-3322dddd7ed9" containerName="payload-pre-processing"
Nov 02 09:55:28.380386 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.380181    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerName="istio-proxy"
Nov 02 09:55:28.380386 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.380194    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebc2cab-97a1-4a35-8f0a-a3d3ea062eb3" containerName="istio-proxy"
Nov 02 09:55:28.380386 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.380205    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b72964-98f0-44cb-9b75-6e79e4c8a17d" containerName="maas-api"
Nov 02 09:55:28.385459 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.385437    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"
Nov 02 09:55:28.389416 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.389346    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-9a69d1\""
Nov 02 09:55:28.390656 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.390547    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-9a69d1-dockercfg-t7rjf\""
Nov 02 09:55:28.396093 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.396071    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"]
Nov 02 09:55:28.422420 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.422391    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"]
Nov 02 09:55:28.427324 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.427300    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"
Nov 02 09:55:28.438242 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.438216    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"]
Nov 02 09:55:28.462703 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.462666    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0089d27-a642-4ce9-8e37-073721510df4-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9\" (UID: \"b0089d27-a642-4ce9-8e37-073721510df4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"
Nov 02 09:55:28.462880 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.462861    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnp5m\" (UniqueName: \"kubernetes.io/projected/b0089d27-a642-4ce9-8e37-073721510df4-kube-api-access-lnp5m\") pod \"payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9\" (UID: \"b0089d27-a642-4ce9-8e37-073721510df4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"
Nov 02 09:55:28.576373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.569469    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnp5m\" (UniqueName: \"kubernetes.io/projected/b0089d27-a642-4ce9-8e37-073721510df4-kube-api-access-lnp5m\") pod \"payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9\" (UID: \"b0089d27-a642-4ce9-8e37-073721510df4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"
Nov 02 09:55:28.576373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.569563    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342c117c-0a54-434c-8a32-8f49498176ab-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c\" (UID: \"342c117c-0a54-434c-8a32-8f49498176ab\") " pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"
Nov 02 09:55:28.576373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.569684    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kcgs\" (UniqueName: \"kubernetes.io/projected/342c117c-0a54-434c-8a32-8f49498176ab-kube-api-access-5kcgs\") pod \"payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c\" (UID: \"342c117c-0a54-434c-8a32-8f49498176ab\") " pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"
Nov 02 09:55:28.576373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.569762    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0089d27-a642-4ce9-8e37-073721510df4-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9\" (UID: \"b0089d27-a642-4ce9-8e37-073721510df4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"
Nov 02 09:55:28.576373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.570803    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0089d27-a642-4ce9-8e37-073721510df4-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9\" (UID: \"b0089d27-a642-4ce9-8e37-073721510df4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"
Nov 02 09:55:28.605378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.604112    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnp5m\" (UniqueName: \"kubernetes.io/projected/b0089d27-a642-4ce9-8e37-073721510df4-kube-api-access-lnp5m\") pod \"payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9\" (UID: \"b0089d27-a642-4ce9-8e37-073721510df4\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"
Nov 02 09:55:28.680459 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.677660    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcgs\" (UniqueName: \"kubernetes.io/projected/342c117c-0a54-434c-8a32-8f49498176ab-kube-api-access-5kcgs\") pod \"payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c\" (UID: \"342c117c-0a54-434c-8a32-8f49498176ab\") " pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"
Nov 02 09:55:28.680459 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.678005    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342c117c-0a54-434c-8a32-8f49498176ab-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c\" (UID: \"342c117c-0a54-434c-8a32-8f49498176ab\") " pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"
Nov 02 09:55:28.680459 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.679026    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342c117c-0a54-434c-8a32-8f49498176ab-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c\" (UID: \"342c117c-0a54-434c-8a32-8f49498176ab\") " pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"
Nov 02 09:55:28.698553 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.697865    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcgs\" (UniqueName: \"kubernetes.io/projected/342c117c-0a54-434c-8a32-8f49498176ab-kube-api-access-5kcgs\") pod \"payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c\" (UID: \"342c117c-0a54-434c-8a32-8f49498176ab\") " pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"
Nov 02 09:55:28.705851 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.701695    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"
Nov 02 09:55:28.748707 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.748649    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"
Nov 02 09:55:28.925090 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.925058    4614 generic.go:356] "Generic (PLEG): container finished" podID="7b48362a-5918-4899-abac-14f674804556" containerID="28e175dbd047d908170c47865844ddc2cc9aa190d5c55b0e61f3d44b6d6da72c" exitCode=0
Nov 02 09:55:28.925272 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:28.925129    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j" event={"ID":"7b48362a-5918-4899-abac-14f674804556","Type":"ContainerDied","Data":"28e175dbd047d908170c47865844ddc2cc9aa190d5c55b0e61f3d44b6d6da72c"}
Nov 02 09:55:29.015806 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.015769    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"]
Nov 02 09:55:29.016728 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:55:29.016690    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0089d27_a642_4ce9_8e37_073721510df4.slice/crio-5d3f04db71d4bb0a8a6ce98650f619b2d2a0ec4e48989605036da51e4387edd8 WatchSource:0}: Error finding container 5d3f04db71d4bb0a8a6ce98650f619b2d2a0ec4e48989605036da51e4387edd8: Status 404 returned error can't find the container with id 5d3f04db71d4bb0a8a6ce98650f619b2d2a0ec4e48989605036da51e4387edd8
Nov 02 09:55:29.048263 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.048240    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:55:29.051193 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.051159    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"]
Nov 02 09:55:29.185654 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.185610    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b48362a-5918-4899-abac-14f674804556-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b48362a-5918-4899-abac-14f674804556-istiod-ca-cert\") pod \"7b48362a-5918-4899-abac-14f674804556\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") "
Nov 02 09:55:29.185830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.185744    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-workload-socket\") pod \"7b48362a-5918-4899-abac-14f674804556\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") "
Nov 02 09:55:29.185830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.185770    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-istio-envoy\") pod \"7b48362a-5918-4899-abac-14f674804556\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") "
Nov 02 09:55:29.185945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.185842    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-credential-socket\") pod \"7b48362a-5918-4899-abac-14f674804556\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") "
Nov 02 09:55:29.185945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.185911    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b48362a-5918-4899-abac-14f674804556-istio-token\" (UniqueName: \"kubernetes.io/projected/7b48362a-5918-4899-abac-14f674804556-istio-token\") pod \"7b48362a-5918-4899-abac-14f674804556\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") "
Nov 02 09:55:29.185945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.185932    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-workload-certs\") pod \"7b48362a-5918-4899-abac-14f674804556\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") "
Nov 02 09:55:29.186129 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.185936    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b48362a-5918-4899-abac-14f674804556-istiod-ca-cert" pod "7b48362a-5918-4899-abac-14f674804556" (UID: "7b48362a-5918-4899-abac-14f674804556"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:29.186129 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.185958    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b48362a-5918-4899-abac-14f674804556-kube-api-access-nbb8p\" (UniqueName: \"kubernetes.io/projected/7b48362a-5918-4899-abac-14f674804556-kube-api-access-nbb8p\") pod \"7b48362a-5918-4899-abac-14f674804556\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") "
Nov 02 09:55:29.186129 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.186007    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b48362a-5918-4899-abac-14f674804556-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b48362a-5918-4899-abac-14f674804556-istio-podinfo\") pod \"7b48362a-5918-4899-abac-14f674804556\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") "
Nov 02 09:55:29.186129 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.186079    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-workload-socket" pod "7b48362a-5918-4899-abac-14f674804556" (UID: "7b48362a-5918-4899-abac-14f674804556"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:29.186129 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.186110    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-istio-data\") pod \"7b48362a-5918-4899-abac-14f674804556\" (UID: \"7b48362a-5918-4899-abac-14f674804556\") "
Nov 02 09:55:29.186407 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.186187    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-credential-socket" pod "7b48362a-5918-4899-abac-14f674804556" (UID: "7b48362a-5918-4899-abac-14f674804556"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:29.186533 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.186508    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-istio-data" pod "7b48362a-5918-4899-abac-14f674804556" (UID: "7b48362a-5918-4899-abac-14f674804556"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:29.186595 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.186539    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-workload-certs" pod "7b48362a-5918-4899-abac-14f674804556" (UID: "7b48362a-5918-4899-abac-14f674804556"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:29.186856 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.186833    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.186959 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.186859    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.186959 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.186869    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.186959 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.186878    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b48362a-5918-4899-abac-14f674804556-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.186959 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.186887    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.188298 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.188241    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-istio-envoy" pod "7b48362a-5918-4899-abac-14f674804556" (UID: "7b48362a-5918-4899-abac-14f674804556"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:29.188390 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.188294    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b48362a-5918-4899-abac-14f674804556-kube-api-access-nbb8p" pod "7b48362a-5918-4899-abac-14f674804556" (UID: "7b48362a-5918-4899-abac-14f674804556"). InnerVolumeSpecName "kube-api-access-nbb8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:29.188618 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.188599    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b48362a-5918-4899-abac-14f674804556-istio-podinfo" pod "7b48362a-5918-4899-abac-14f674804556" (UID: "7b48362a-5918-4899-abac-14f674804556"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:55:29.188678 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.188623    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b48362a-5918-4899-abac-14f674804556-istio-token" pod "7b48362a-5918-4899-abac-14f674804556" (UID: "7b48362a-5918-4899-abac-14f674804556"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:29.287897 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.287867    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b48362a-5918-4899-abac-14f674804556-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.287897 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.287894    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b48362a-5918-4899-abac-14f674804556-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.287897 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.287904    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbb8p\" (UniqueName: \"kubernetes.io/projected/7b48362a-5918-4899-abac-14f674804556-kube-api-access-nbb8p\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.288134 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.287914    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b48362a-5918-4899-abac-14f674804556-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:29.938927 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.937926    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9" event={"ID":"b0089d27-a642-4ce9-8e37-073721510df4","Type":"ContainerStarted","Data":"76c4c0fa3342e6f17cda185b885e1ed8734b36132a436f7758449c559d957dfa"}
Nov 02 09:55:29.938927 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.937983    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9" event={"ID":"b0089d27-a642-4ce9-8e37-073721510df4","Type":"ContainerStarted","Data":"5d3f04db71d4bb0a8a6ce98650f619b2d2a0ec4e48989605036da51e4387edd8"}
Nov 02 09:55:29.938927 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.938653    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"
Nov 02 09:55:29.942393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.941857    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c" event={"ID":"342c117c-0a54-434c-8a32-8f49498176ab","Type":"ContainerStarted","Data":"9dc3e970d06a37e4286940832119b2c401116c42da6afa5da2efdf9bf1f26ba2"}
Nov 02 09:55:29.942393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.941905    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c" event={"ID":"342c117c-0a54-434c-8a32-8f49498176ab","Type":"ContainerStarted","Data":"5cff45e4a00314adb407556eb12e641080c29d655ddd6fa9ad2a04b04c84ed8b"}
Nov 02 09:55:29.942892 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.942828    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"
Nov 02 09:55:29.944816 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.944685    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j" event={"ID":"7b48362a-5918-4899-abac-14f674804556","Type":"ContainerDied","Data":"5816cc045c68ed0e387767065118bd07bd5c9eba1698ea69a5a23617be473b5b"}
Nov 02 09:55:29.944816 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.944749    4614 scope.go:122] "RemoveContainer" containerID="28e175dbd047d908170c47865844ddc2cc9aa190d5c55b0e61f3d44b6d6da72c"
Nov 02 09:55:29.945564 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.945506    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"
Nov 02 09:55:29.961136 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.961090    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9" podStartSLOduration=1.96107588 podStartE2EDuration="1.96107588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:29.958932324 +0000 UTC m=+1210.791590404" watchObservedRunningTime="2026-11-02 09:55:29.96107588 +0000 UTC m=+1210.793733951"
Nov 02 09:55:29.975467 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.975435    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"]
Nov 02 09:55:29.979704 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:29.979676    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9cbd331c-openshift-default-574b7ff47c-6hx9j"]
Nov 02 09:55:30.003208 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:30.003099    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c" podStartSLOduration=2.003085901 podStartE2EDuration="2.003085901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:30.001503158 +0000 UTC m=+1210.834161233" watchObservedRunningTime="2026-11-02 09:55:30.003085901 +0000 UTC m=+1210.835743980"
Nov 02 09:55:31.375376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:31.373668    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b48362a-5918-4899-abac-14f674804556" path="/var/lib/kubelet/pods/7b48362a-5918-4899-abac-14f674804556/volumes"
Nov 02 09:55:35.706021 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.705977    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"]
Nov 02 09:55:35.706586 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.706569    4614 cpu_manager.go:414] "containerMap: removing container" podUID="7b48362a-5918-4899-abac-14f674804556" containerName="istio-proxy"
Nov 02 09:55:35.706586 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.706586    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b48362a-5918-4899-abac-14f674804556" containerName="istio-proxy"
Nov 02 09:55:35.706717 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.706706    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b48362a-5918-4899-abac-14f674804556" containerName="istio-proxy"
Nov 02 09:55:35.711767 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.711744    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:35.723056 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.723019    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-9a69d1-serving-cert\""
Nov 02 09:55:35.759265 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.759230    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"]
Nov 02 09:55:35.760431 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.760399    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e4deca-3ded-4b46-983c-911a324f8eaa-maas-api-tls\") pod \"maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:35.760558 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.760458    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e4deca-3ded-4b46-983c-911a324f8eaa-metrics-certs\") pod \"maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:35.760621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.760570    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbjnt\" (UniqueName: \"kubernetes.io/projected/29e4deca-3ded-4b46-983c-911a324f8eaa-kube-api-access-nbjnt\") pod \"maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:35.760706 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.760689    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e4deca-3ded-4b46-983c-911a324f8eaa-openshift-service-ca\") pod \"maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:35.862529 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.862494    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e4deca-3ded-4b46-983c-911a324f8eaa-maas-api-tls\") pod \"maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:35.862701 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.862544    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e4deca-3ded-4b46-983c-911a324f8eaa-metrics-certs\") pod \"maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:35.862701 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.862575    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjnt\" (UniqueName: \"kubernetes.io/projected/29e4deca-3ded-4b46-983c-911a324f8eaa-kube-api-access-nbjnt\") pod \"maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:35.862701 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.862647    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e4deca-3ded-4b46-983c-911a324f8eaa-openshift-service-ca\") pod \"maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:35.863441 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.863413    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e4deca-3ded-4b46-983c-911a324f8eaa-openshift-service-ca\") pod \"maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:35.864952 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.864929    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e4deca-3ded-4b46-983c-911a324f8eaa-maas-api-tls\") pod \"maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:35.865122 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.865100    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e4deca-3ded-4b46-983c-911a324f8eaa-metrics-certs\") pod \"maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:35.890385 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:35.875297    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjnt\" (UniqueName: \"kubernetes.io/projected/29e4deca-3ded-4b46-983c-911a324f8eaa-kube-api-access-nbjnt\") pod \"maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:36.067789 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:36.067756    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:36.488726 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:55:36.485331    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e4deca_3ded_4b46_983c_911a324f8eaa.slice/crio-5c9e18ff5ab1756a23047f47c73f5418cc6c3a5fa769927c012d2c9ea332e1b6 WatchSource:0}: Error finding container 5c9e18ff5ab1756a23047f47c73f5418cc6c3a5fa769927c012d2c9ea332e1b6: Status 404 returned error can't find the container with id 5c9e18ff5ab1756a23047f47c73f5418cc6c3a5fa769927c012d2c9ea332e1b6
Nov 02 09:55:36.497986 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:36.497176    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"]
Nov 02 09:55:37.084068 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:37.084032    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn" event={"ID":"29e4deca-3ded-4b46-983c-911a324f8eaa","Type":"ContainerStarted","Data":"31f6ede88deaca2e41bf0fb77d0ff2314317f954b4b3b16b44176c12fc732087"}
Nov 02 09:55:37.084478 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:37.084088    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn" event={"ID":"29e4deca-3ded-4b46-983c-911a324f8eaa","Type":"ContainerStarted","Data":"5c9e18ff5ab1756a23047f47c73f5418cc6c3a5fa769927c012d2c9ea332e1b6"}
Nov 02 09:55:37.084478 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:37.084177    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:37.108857 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:37.108797    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn" podStartSLOduration=1.62347554 podStartE2EDuration="2.108779663s" totalImagesPullingTime="485.304123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:37.10707284 +0000 UTC m=+1217.939730920" watchObservedRunningTime="2026-11-02 09:55:37.108779663 +0000 UTC m=+1217.941437744"
Nov 02 09:55:37.933430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:37.914418    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"]
Nov 02 09:55:37.963374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:37.957681    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.179382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.177771    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.179382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.177971    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.179382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.178231    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.188047 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.187971    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.188388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.188329    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.188695 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.188674    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.188898 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.188878    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.189124 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.189104    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x79mx\" (UniqueName: \"kubernetes.io/projected/f0c328ac-256e-4e52-bb7b-81dfac9df62f-kube-api-access-x79mx\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.192745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.189340    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.257450 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.257412    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"]
Nov 02 09:55:38.302570 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.302535    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.304454 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.304427    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.304683 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.304663    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.305281 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.305263    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.305801 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.305695    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.305975 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.305957    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.306379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.306308    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.306546 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.306528    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.307380 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.307285    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.307527 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.307511    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.307669 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.307652    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.307799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.307783    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x79mx\" (UniqueName: \"kubernetes.io/projected/f0c328ac-256e-4e52-bb7b-81dfac9df62f-kube-api-access-x79mx\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.307996 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.307977    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.308658 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.308638    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.321419 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.321393    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"]
Nov 02 09:55:38.336567 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.330632    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.425591 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.409960    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.425591 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.410194    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.425591 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.410229    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.425591 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.410260    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.425591 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.410335    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.425591 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.424219    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"]
Nov 02 09:55:38.426508 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.426478    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/125ab3c0-c75a-4866-bbfd-d490769e6146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.426660 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.426541    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9f8g\" (UniqueName: \"kubernetes.io/projected/125ab3c0-c75a-4866-bbfd-d490769e6146-kube-api-access-b9f8g\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.426660 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.426638    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.426836 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.426709    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.467210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.466157    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.467210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.466467    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.467210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.467107    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.500428 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.489121    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x79mx\" (UniqueName: \"kubernetes.io/projected/f0c328ac-256e-4e52-bb7b-81dfac9df62f-kube-api-access-x79mx\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-wdc6m\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.500428 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.493618    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:38.543278 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.543236    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.546639 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.546606    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/125ab3c0-c75a-4866-bbfd-d490769e6146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.546841 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.546819    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9f8g\" (UniqueName: \"kubernetes.io/projected/125ab3c0-c75a-4866-bbfd-d490769e6146-kube-api-access-b9f8g\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.547034 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.547016    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.547168 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.547151    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.547309 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.547291    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.547627 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.547607    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.547768 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.547752    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.547879 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.547862    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.548910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.548889    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.549707 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.549682    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/125ab3c0-c75a-4866-bbfd-d490769e6146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.550325 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.550303    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.550778 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.550756    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.551140 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.551121    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.647373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.633249    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9f8g\" (UniqueName: \"kubernetes.io/projected/125ab3c0-c75a-4866-bbfd-d490769e6146-kube-api-access-b9f8g\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.647373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.636025    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.647373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.637869    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.661387 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.659653    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-zjvjd\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:38.720431 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:38.717616    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:39.909305 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:39.909235    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:55:39.959107 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:39.959009    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"]
Nov 02 09:55:39.962446 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:55:39.962410    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125ab3c0_c75a_4866_bbfd_d490769e6146.slice/crio-dd080ac9d37c8f66479b21be6edf3c69bca7dad78c0df2f71f636aac75bae9d7 WatchSource:0}: Error finding container dd080ac9d37c8f66479b21be6edf3c69bca7dad78c0df2f71f636aac75bae9d7: Status 404 returned error can't find the container with id dd080ac9d37c8f66479b21be6edf3c69bca7dad78c0df2f71f636aac75bae9d7
Nov 02 09:55:39.966563 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:39.965975    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:39.966563 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:39.966075    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:39.966563 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:39.966122    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:39.995525 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:39.995498    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"]
Nov 02 09:55:39.997446 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:55:39.997414    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c328ac_256e_4e52_bb7b_81dfac9df62f.slice/crio-c5576212b8521690393acb53f447e798d4f68b8d528963a8a762672bfdc2bedb WatchSource:0}: Error finding container c5576212b8521690393acb53f447e798d4f68b8d528963a8a762672bfdc2bedb: Status 404 returned error can't find the container with id c5576212b8521690393acb53f447e798d4f68b8d528963a8a762672bfdc2bedb
Nov 02 09:55:40.000719 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:40.000671    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:40.000818 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:40.000774    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:40.000865 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:40.000820    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:40.401939 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:40.401889    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" event={"ID":"125ab3c0-c75a-4866-bbfd-d490769e6146","Type":"ContainerStarted","Data":"619c08182293544fa8052b92d33463817c457f4fe02b637526beee3d12a216e6"}
Nov 02 09:55:40.402191 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:40.401951    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" event={"ID":"125ab3c0-c75a-4866-bbfd-d490769e6146","Type":"ContainerStarted","Data":"dd080ac9d37c8f66479b21be6edf3c69bca7dad78c0df2f71f636aac75bae9d7"}
Nov 02 09:55:40.405714 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:40.405684    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" event={"ID":"f0c328ac-256e-4e52-bb7b-81dfac9df62f","Type":"ContainerStarted","Data":"4f24b79772502bbe789ec2c9a215d33b16fc1fe37d00eab00950d645b7f11372"}
Nov 02 09:55:40.405923 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:40.405738    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" event={"ID":"f0c328ac-256e-4e52-bb7b-81dfac9df62f","Type":"ContainerStarted","Data":"c5576212b8521690393acb53f447e798d4f68b8d528963a8a762672bfdc2bedb"}
Nov 02 09:55:40.427412 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:40.425676    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" podStartSLOduration=3.425656511 podStartE2EDuration="3.425656511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:40.425169949 +0000 UTC m=+1221.257828027" watchObservedRunningTime="2026-11-02 09:55:40.425656511 +0000 UTC m=+1221.258314591"
Nov 02 09:55:40.495430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:40.495345    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:40.752374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:40.744571    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:40.980404 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:40.979949    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"
Nov 02 09:55:41.000621 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:41.000572    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" podStartSLOduration=4.000557024 podStartE2EDuration="4.000557024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:40.455516959 +0000 UTC m=+1221.288175039" watchObservedRunningTime="2026-11-02 09:55:41.000557024 +0000 UTC m=+1221.833215104"
Nov 02 09:55:41.496477 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:41.496435    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 09:55:41.496652 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:41.496506    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 09:55:41.655460 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:41.655416    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7366-5b297b"]
Nov 02 09:55:41.661105 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:41.661077    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-5b297b"
Nov 02 09:55:41.675373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:41.667616    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x8bqk\""
Nov 02 09:55:41.688081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:41.688050    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7366-5b297b"]
Nov 02 09:55:41.720708 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:41.720676    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgc7m\" (UniqueName: \"kubernetes.io/projected/c42c3bb6-1061-48f3-a318-7b44e3fb0b52-kube-api-access-kgc7m\") pod \"test-curl-7366-5b297b\" (UID: \"c42c3bb6-1061-48f3-a318-7b44e3fb0b52\") " pod="opendatahub/test-curl-7366-5b297b"
Nov 02 09:55:41.757439 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:41.756732    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 09:55:41.757439 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:41.756799    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 09:55:41.847376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:41.833112    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgc7m\" (UniqueName: \"kubernetes.io/projected/c42c3bb6-1061-48f3-a318-7b44e3fb0b52-kube-api-access-kgc7m\") pod \"test-curl-7366-5b297b\" (UID: \"c42c3bb6-1061-48f3-a318-7b44e3fb0b52\") " pod="opendatahub/test-curl-7366-5b297b"
Nov 02 09:55:41.893375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:41.886809    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgc7m\" (UniqueName: \"kubernetes.io/projected/c42c3bb6-1061-48f3-a318-7b44e3fb0b52-kube-api-access-kgc7m\") pod \"test-curl-7366-5b297b\" (UID: \"c42c3bb6-1061-48f3-a318-7b44e3fb0b52\") " pod="opendatahub/test-curl-7366-5b297b"
Nov 02 09:55:41.982376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:41.981889    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"
Nov 02 09:55:41.998969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:41.998938    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-5b297b"
Nov 02 09:55:42.406774 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:42.406747    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7366-5b297b"]
Nov 02 09:55:42.407933 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:55:42.407908    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc42c3bb6_1061_48f3_a318_7b44e3fb0b52.slice/crio-a4b12ae7c37ebd2e7f078300e957cb45d545a29124146ba414d1b79804d08e88 WatchSource:0}: Error finding container a4b12ae7c37ebd2e7f078300e957cb45d545a29124146ba414d1b79804d08e88: Status 404 returned error can't find the container with id a4b12ae7c37ebd2e7f078300e957cb45d545a29124146ba414d1b79804d08e88
Nov 02 09:55:42.421341 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:42.421309    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7366-5b297b" event={"ID":"c42c3bb6-1061-48f3-a318-7b44e3fb0b52","Type":"ContainerStarted","Data":"a4b12ae7c37ebd2e7f078300e957cb45d545a29124146ba414d1b79804d08e88"}
Nov 02 09:55:42.494955 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:42.494932    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 09:55:42.495078 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:42.494974    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 09:55:42.744205 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:42.743756    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 09:55:42.744205 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:42.743821    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 09:55:43.175609 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:43.175527    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:55:43.487565 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:43.487525    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7366-5b297b" event={"ID":"c42c3bb6-1061-48f3-a318-7b44e3fb0b52","Type":"ContainerStarted","Data":"c58ce147745eb660d5e384f5dd0fb1198b477e0c9c002d5f2a3c839571b78128"}
Nov 02 09:55:43.494374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:43.494333    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 09:55:43.494538 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:43.494402    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 09:55:43.506976 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:43.506709    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7366-5b297b" podStartSLOduration=2.506691236 podStartE2EDuration="2.506691236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:43.503372849 +0000 UTC m=+1224.336030931" watchObservedRunningTime="2026-11-02 09:55:43.506691236 +0000 UTC m=+1224.339349317"
Nov 02 09:55:43.676948 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:43.643911    4614 patch_prober.go:29] interesting pod/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:55:43.676948 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:43.676487    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:43.754432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:43.753733    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 09:55:43.754432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:43.753793    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 09:55:44.303217 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.303178    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7366-5b297b"]
Nov 02 09:55:44.309257 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.309237    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7366-5b297b"]
Nov 02 09:55:44.481613 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.481573    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7366-ae3ae1"]
Nov 02 09:55:44.482731 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.482690    4614 cpu_manager.go:414] "containerMap: removing container" podUID="c42c3bb6-1061-48f3-a318-7b44e3fb0b52" containerName="test-curl-7366-5b297b"
Nov 02 09:55:44.482731 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.482726    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42c3bb6-1061-48f3-a318-7b44e3fb0b52" containerName="test-curl-7366-5b297b"
Nov 02 09:55:44.483925 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.483899    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42c3bb6-1061-48f3-a318-7b44e3fb0b52" containerName="test-curl-7366-5b297b"
Nov 02 09:55:44.496711 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.496671    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 09:55:44.496841 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.496755    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 09:55:44.505486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.505458    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7366-ae3ae1"]
Nov 02 09:55:44.505593 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.505556    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:44.687319 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.687236    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwnhc\" (UniqueName: \"kubernetes.io/projected/b124f6e5-c41f-4b7b-a0b6-7979913e9eb2-kube-api-access-rwnhc\") pod \"test-curl-7366-ae3ae1\" (UID: \"b124f6e5-c41f-4b7b-a0b6-7979913e9eb2\") " pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:44.753009 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.749471    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 09:55:44.753009 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.749529    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 09:55:44.789165 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.789025    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwnhc\" (UniqueName: \"kubernetes.io/projected/b124f6e5-c41f-4b7b-a0b6-7979913e9eb2-kube-api-access-rwnhc\") pod \"test-curl-7366-ae3ae1\" (UID: \"b124f6e5-c41f-4b7b-a0b6-7979913e9eb2\") " pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:44.801180 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.801146    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwnhc\" (UniqueName: \"kubernetes.io/projected/b124f6e5-c41f-4b7b-a0b6-7979913e9eb2-kube-api-access-rwnhc\") pod \"test-curl-7366-ae3ae1\" (UID: \"b124f6e5-c41f-4b7b-a0b6-7979913e9eb2\") " pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:44.821519 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:44.821487    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:45.611427 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:45.600202    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 09:55:45.611427 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:45.600269    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 09:55:45.631809 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:45.622924    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7366-ae3ae1"]
Nov 02 09:55:45.650863 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:45.650807    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7366-5b297b" podUID="c42c3bb6-1061-48f3-a318-7b44e3fb0b52" containerName="test-curl-7366-5b297b" containerID="cri-o://c58ce147745eb660d5e384f5dd0fb1198b477e0c9c002d5f2a3c839571b78128" gracePeriod=2
Nov 02 09:55:45.708505 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:55:45.708462    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb124f6e5_c41f_4b7b_a0b6_7979913e9eb2.slice/crio-669b89664b14cf6c31d37010dba7d45bf8d8c844ef5cbf35676a612bb1dba624 WatchSource:0}: Error finding container 669b89664b14cf6c31d37010dba7d45bf8d8c844ef5cbf35676a612bb1dba624: Status 404 returned error can't find the container with id 669b89664b14cf6c31d37010dba7d45bf8d8c844ef5cbf35676a612bb1dba624
Nov 02 09:55:45.749746 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:45.736133    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 09:55:45.749746 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:45.736191    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 09:55:46.498429 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:46.497775    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 09:55:46.498429 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:46.497840    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 09:55:46.656774 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:46.656732    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7366-ae3ae1" event={"ID":"b124f6e5-c41f-4b7b-a0b6-7979913e9eb2","Type":"ContainerStarted","Data":"1d8c666df1ff7be9451a9580781aed84d6b7074a54443746652e124bd477d4f2"}
Nov 02 09:55:46.657282 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:46.656787    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7366-ae3ae1" event={"ID":"b124f6e5-c41f-4b7b-a0b6-7979913e9eb2","Type":"ContainerStarted","Data":"669b89664b14cf6c31d37010dba7d45bf8d8c844ef5cbf35676a612bb1dba624"}
Nov 02 09:55:46.658704 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:46.658677    4614 generic.go:356] "Generic (PLEG): container finished" podID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerID="d67c97639598c40189817f3e3d6897e3450c7f73ae1b62758a96115850cd2fad" exitCode=0
Nov 02 09:55:46.658855 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:46.658751    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" event={"ID":"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2","Type":"ContainerDied","Data":"d67c97639598c40189817f3e3d6897e3450c7f73ae1b62758a96115850cd2fad"}
Nov 02 09:55:46.659161 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:46.659147    4614 scope.go:122] "RemoveContainer" containerID="d67c97639598c40189817f3e3d6897e3450c7f73ae1b62758a96115850cd2fad"
Nov 02 09:55:46.660924 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:46.660885    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:46.661051 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:46.660956    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:46.661051 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:46.661006    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:55:46.674799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:46.674753    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7366-ae3ae1" podStartSLOduration=2.67473906 podStartE2EDuration="2.67473906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:46.674262613 +0000 UTC m=+1227.506920694" watchObservedRunningTime="2026-11-02 09:55:46.67473906 +0000 UTC m=+1227.507397173"
Nov 02 09:55:46.734383 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:46.733677    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 09:55:46.734383 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:46.733735    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 09:55:47.495085 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:47.495050    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 09:55:47.495295 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:47.495104    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 09:55:47.570915 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:47.550405    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7366-ae3ae1"]
Nov 02 09:55:47.570915 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:47.568492    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7366-ae3ae1"]
Nov 02 09:55:47.673247 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:47.673207    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" event={"ID":"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2","Type":"ContainerStarted","Data":"70a395e4965b826ea7095eb544d67a3f752ed3769c242bd70c700f6b9fd1f994"}
Nov 02 09:55:47.777432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:47.735745    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 09:55:47.777432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:47.735804    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 09:55:48.486611 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.486577    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-5b297b"
Nov 02 09:55:48.495964 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.495934    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 09:55:48.496197 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.496169    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 09:55:48.564377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.564234    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42c3bb6-1061-48f3-a318-7b44e3fb0b52-kube-api-access-kgc7m\" (UniqueName: \"kubernetes.io/projected/c42c3bb6-1061-48f3-a318-7b44e3fb0b52-kube-api-access-kgc7m\") pod \"c42c3bb6-1061-48f3-a318-7b44e3fb0b52\" (UID: \"c42c3bb6-1061-48f3-a318-7b44e3fb0b52\") "
Nov 02 09:55:48.568998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.568964    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42c3bb6-1061-48f3-a318-7b44e3fb0b52-kube-api-access-kgc7m" pod "c42c3bb6-1061-48f3-a318-7b44e3fb0b52" (UID: "c42c3bb6-1061-48f3-a318-7b44e3fb0b52"). InnerVolumeSpecName "kube-api-access-kgc7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:48.586776 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.586744    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:48.587494 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.587463    4614 patch_prober.go:29] interesting pod/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 09:55:48.587614 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.587517    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 09:55:48.672154 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.672116    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgc7m\" (UniqueName: \"kubernetes.io/projected/c42c3bb6-1061-48f3-a318-7b44e3fb0b52-kube-api-access-kgc7m\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:48.685851 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.685816    4614 generic.go:356] "Generic (PLEG): container finished" podID="c42c3bb6-1061-48f3-a318-7b44e3fb0b52" containerID="c58ce147745eb660d5e384f5dd0fb1198b477e0c9c002d5f2a3c839571b78128" exitCode=137
Nov 02 09:55:48.686379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.685915    4614 scope.go:122] "RemoveContainer" containerID="c58ce147745eb660d5e384f5dd0fb1198b477e0c9c002d5f2a3c839571b78128"
Nov 02 09:55:48.686379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.686255    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7366-ae3ae1" podUID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" containerName="test-curl-7366-ae3ae1" containerID="cri-o://1d8c666df1ff7be9451a9580781aed84d6b7074a54443746652e124bd477d4f2" gracePeriod=2
Nov 02 09:55:48.687732 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.686752    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-5b297b"
Nov 02 09:55:48.688905 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:48.688873    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-ae3ae1\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:48.691255 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:48.691197    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-ae3ae1\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:48.701394 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.701374    4614 scope.go:122] "RemoveContainer" containerID="c58ce147745eb660d5e384f5dd0fb1198b477e0c9c002d5f2a3c839571b78128"
Nov 02 09:55:48.701890 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:48.701864    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58ce147745eb660d5e384f5dd0fb1198b477e0c9c002d5f2a3c839571b78128\": container with ID starting with c58ce147745eb660d5e384f5dd0fb1198b477e0c9c002d5f2a3c839571b78128 not found: ID does not exist" containerID="c58ce147745eb660d5e384f5dd0fb1198b477e0c9c002d5f2a3c839571b78128"
Nov 02 09:55:48.701961 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.701905    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58ce147745eb660d5e384f5dd0fb1198b477e0c9c002d5f2a3c839571b78128"} err="failed to get container status \"c58ce147745eb660d5e384f5dd0fb1198b477e0c9c002d5f2a3c839571b78128\": rpc error: code = NotFound desc = could not find container \"c58ce147745eb660d5e384f5dd0fb1198b477e0c9c002d5f2a3c839571b78128\": container with ID starting with c58ce147745eb660d5e384f5dd0fb1198b477e0c9c002d5f2a3c839571b78128 not found: ID does not exist"
Nov 02 09:55:48.703540 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:48.702258    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-5b297b\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="c42c3bb6-1061-48f3-a318-7b44e3fb0b52" pod="opendatahub/test-curl-7366-5b297b"
Nov 02 09:55:48.704514 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:48.704493    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-ae3ae1\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:48.736375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.734316    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 09:55:48.736375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:48.734394    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 09:55:49.317792 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:49.317745    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-ae3ae1\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:49.318748 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:49.318723    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c42c3bb6-1061-48f3-a318-7b44e3fb0b52" path="/var/lib/kubelet/pods/c42c3bb6-1061-48f3-a318-7b44e3fb0b52/volumes"
Nov 02 09:55:49.319990 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:49.319959    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-5b297b\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="c42c3bb6-1061-48f3-a318-7b44e3fb0b52" pod="opendatahub/test-curl-7366-5b297b"
Nov 02 09:55:49.495271 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:49.495237    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 09:55:49.495444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:49.495290    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 09:55:49.587169 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:49.587069    4614 patch_prober.go:29] interesting pod/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 09:55:49.587169 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:49.587136    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 09:55:49.736737 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:49.736688    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 02 09:55:49.737333 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:49.737302    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 02 09:55:50.587306 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:50.587269    4614 patch_prober.go:29] interesting pod/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 09:55:50.587634 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:50.587600    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 09:55:51.082910 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:51.082869    4614 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb124f6e5_c41f_4b7b_a0b6_7979913e9eb2.slice/crio-conmon-1d8c666df1ff7be9451a9580781aed84d6b7074a54443746652e124bd477d4f2.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 09:55:51.384189 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:51.384158    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:51.405844 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:51.405755    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b124f6e5-c41f-4b7b-a0b6-7979913e9eb2-kube-api-access-rwnhc\" (UniqueName: \"kubernetes.io/projected/b124f6e5-c41f-4b7b-a0b6-7979913e9eb2-kube-api-access-rwnhc\") pod \"b124f6e5-c41f-4b7b-a0b6-7979913e9eb2\" (UID: \"b124f6e5-c41f-4b7b-a0b6-7979913e9eb2\") "
Nov 02 09:55:51.408909 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:51.408876    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b124f6e5-c41f-4b7b-a0b6-7979913e9eb2-kube-api-access-rwnhc" pod "b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" (UID: "b124f6e5-c41f-4b7b-a0b6-7979913e9eb2"). InnerVolumeSpecName "kube-api-access-rwnhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:55:51.494784 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:51.494739    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:55:51.495005 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:51.494835    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:51.508596 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:51.508550    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwnhc\" (UniqueName: \"kubernetes.io/projected/b124f6e5-c41f-4b7b-a0b6-7979913e9eb2-kube-api-access-rwnhc\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:55:51.704608 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:51.704089    4614 patch_prober.go:29] interesting pod/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 09:55:51.704608 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:51.704149    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 09:55:51.781529 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:51.781494    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:55:51.781853 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:51.781805    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:51.808526 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:51.800601    4614 generic.go:356] "Generic (PLEG): container finished" podID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" containerID="1d8c666df1ff7be9451a9580781aed84d6b7074a54443746652e124bd477d4f2" exitCode=137
Nov 02 09:55:51.808957 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:51.808935    4614 scope.go:122] "RemoveContainer" containerID="1d8c666df1ff7be9451a9580781aed84d6b7074a54443746652e124bd477d4f2"
Nov 02 09:55:51.809219 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:51.809196    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:51.875629 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:51.875576    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-ae3ae1\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:52.094599 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:52.092435    4614 scope.go:122] "RemoveContainer" containerID="1d8c666df1ff7be9451a9580781aed84d6b7074a54443746652e124bd477d4f2"
Nov 02 09:55:52.101393 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:52.100178    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8c666df1ff7be9451a9580781aed84d6b7074a54443746652e124bd477d4f2\": container with ID starting with 1d8c666df1ff7be9451a9580781aed84d6b7074a54443746652e124bd477d4f2 not found: ID does not exist" containerID="1d8c666df1ff7be9451a9580781aed84d6b7074a54443746652e124bd477d4f2"
Nov 02 09:55:52.101393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:52.100218    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8c666df1ff7be9451a9580781aed84d6b7074a54443746652e124bd477d4f2"} err="failed to get container status \"1d8c666df1ff7be9451a9580781aed84d6b7074a54443746652e124bd477d4f2\": rpc error: code = NotFound desc = could not find container \"1d8c666df1ff7be9451a9580781aed84d6b7074a54443746652e124bd477d4f2\": container with ID starting with 1d8c666df1ff7be9451a9580781aed84d6b7074a54443746652e124bd477d4f2 not found: ID does not exist"
Nov 02 09:55:52.106583 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:52.103184    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:52.106583 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:52.105218    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:52.109113 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:52.109007    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 09:55:52.250786 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:52.249619    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-ae3ae1\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:52.386196 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:52.386091    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-ae3ae1\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:52.670878 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:52.634978    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:52.670878 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:52.640306    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:52.700074 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:52.700041    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 09:55:52.708394 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:52.700890    4614 patch_prober.go:29] interesting pod/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 09:55:52.760658 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:52.760607    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 09:55:52.761621 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:52.761577    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-ae3ae1\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:52.769174 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:55:52.769135    4614 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7366-ae3ae1\" is forbidden: User \"system:node:ip-10-0-134-174.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-174.us-east-2.compute.internal' and this object" podUID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" pod="opendatahub/test-curl-7366-ae3ae1"
Nov 02 09:55:53.377170 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:53.377131    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" path="/var/lib/kubelet/pods/b124f6e5-c41f-4b7b-a0b6-7979913e9eb2/volumes"
Nov 02 09:55:53.596500 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:53.596469    4614 patch_prober.go:29] interesting pod/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 09:55:53.596804 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:53.596776    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 09:55:53.737433 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:53.737265    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:53.744576 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:53.744526    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"]
Nov 02 09:55:53.745116 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:53.745074    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerName="istio-proxy" containerID="cri-o://bc0c235faea8ffbaaef6f6b3f9b140001b1e19f6db600f446f52bdde6ae4f061" gracePeriod=30
Nov 02 09:55:53.981487 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:53.981440    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:54.046600 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.046499    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:54.320064 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.319982    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"]
Nov 02 09:55:54.321156 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.321128    4614 cpu_manager.go:414] "containerMap: removing container" podUID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" containerName="test-curl-7366-ae3ae1"
Nov 02 09:55:54.321318 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.321304    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" containerName="test-curl-7366-ae3ae1"
Nov 02 09:55:54.321671 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.321654    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="b124f6e5-c41f-4b7b-a0b6-7979913e9eb2" containerName="test-curl-7366-ae3ae1"
Nov 02 09:55:54.366924 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.366889    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"]
Nov 02 09:55:54.367224 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.367207    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.387001 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.386972    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-9a69d1\"/\"kube-root-ca.crt\""
Nov 02 09:55:54.388543 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.388512    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-9a69d1\"/\"explicit-model-9a69d1-kserve-self-signed-certs\""
Nov 02 09:55:54.388892 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.388874    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-9a69d1\"/\"openshift-service-ca.crt\""
Nov 02 09:55:54.389122 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.389108    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-9a69d1\"/\"default-dockercfg-xwpnm\""
Nov 02 09:55:54.466690 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.466560    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-home\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.489814 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.489064    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vqcc\" (UniqueName: \"kubernetes.io/projected/75f06396-128b-4121-af72-94df8d58bc59-kube-api-access-4vqcc\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.490228 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.490206    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f06396-128b-4121-af72-94df8d58bc59-tls-certs\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.490980 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.490961    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-model-cache\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.491507 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.491487    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-dshm\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.492094 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.492067    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-tmp-dir\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.599331 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.599232    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-tmp-dir\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.599670 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.599645    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-home\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.599913 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.599893    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vqcc\" (UniqueName: \"kubernetes.io/projected/75f06396-128b-4121-af72-94df8d58bc59-kube-api-access-4vqcc\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.600051 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.600036    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f06396-128b-4121-af72-94df8d58bc59-tls-certs\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.600206 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.600190    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-model-cache\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.600328 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.600314    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-dshm\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.601092 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.601013    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-model-cache\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.601092 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.599975    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-home\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.601285 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.599749    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-tmp-dir\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.631427 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.627079    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f06396-128b-4121-af72-94df8d58bc59-tls-certs\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.632135 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.632080    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vqcc\" (UniqueName: \"kubernetes.io/projected/75f06396-128b-4121-af72-94df8d58bc59-kube-api-access-4vqcc\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.632587 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.632545    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-dshm\") pod \"explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") " pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:54.762872 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:54.762835    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:55.638393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:55.637477    4614 patch_prober.go:29] interesting pod/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:55:55.638393 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:55.637545    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:55.700049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:55.699903    4614 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Nov 02 09:55:55.835382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:55.833019    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:55.865094 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:55.836751    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:55.865094 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:55.852986    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:55:55.950421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:55.943164    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"]
Nov 02 09:55:55.964871 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:55:55.964843    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75f06396_128b_4121_af72_94df8d58bc59.slice/crio-a3cf2785adc50c286fdb6e6b8f2a6b23b0485ffc93a606bcceed3fde051a5a35 WatchSource:0}: Error finding container a3cf2785adc50c286fdb6e6b8f2a6b23b0485ffc93a606bcceed3fde051a5a35: Status 404 returned error can't find the container with id a3cf2785adc50c286fdb6e6b8f2a6b23b0485ffc93a606bcceed3fde051a5a35
Nov 02 09:55:56.430719 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:56.428505    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb" event={"ID":"75f06396-128b-4121-af72-94df8d58bc59","Type":"ContainerStarted","Data":"4780149ee7bc98533a40ab19cfd9a7147ae7974e0844a7cb293e27d70bd39303"}
Nov 02 09:55:56.430719 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:56.428576    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb" event={"ID":"75f06396-128b-4121-af72-94df8d58bc59","Type":"ContainerStarted","Data":"a3cf2785adc50c286fdb6e6b8f2a6b23b0485ffc93a606bcceed3fde051a5a35"}
Nov 02 09:55:56.502296 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:56.502230    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb" podStartSLOduration=2.502209722 podStartE2EDuration="2.502209722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:55:56.453696406 +0000 UTC m=+1237.286354487" watchObservedRunningTime="2026-11-02 09:55:56.502209722 +0000 UTC m=+1237.334867804"
Nov 02 09:55:56.788374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:56.782465    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:55:58.049024 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:58.048974    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Nov 02 09:55:58.273375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:58.272995    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:58.548095 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:58.547502    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:59.652499 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.648738    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Nov 02 09:55:59.654893 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.654869    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:55:59.682006 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.681325    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:55:59.727815 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.722940    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:55:59.897421 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.897260    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:55:59.917409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.915627    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-workload-socket\") pod \"bac35651-e552-4e21-89c9-c76ee6d72ec4\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") "
Nov 02 09:55:59.917409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.915679    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-credential-socket\") pod \"bac35651-e552-4e21-89c9-c76ee6d72ec4\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") "
Nov 02 09:55:59.917409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.915710    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-envoy\") pod \"bac35651-e552-4e21-89c9-c76ee6d72ec4\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") "
Nov 02 09:55:59.917409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.915767    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac35651-e552-4e21-89c9-c76ee6d72ec4-kube-api-access-5xxmb\" (UniqueName: \"kubernetes.io/projected/bac35651-e552-4e21-89c9-c76ee6d72ec4-kube-api-access-5xxmb\") pod \"bac35651-e552-4e21-89c9-c76ee6d72ec4\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") "
Nov 02 09:55:59.917409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.915908    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac35651-e552-4e21-89c9-c76ee6d72ec4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac35651-e552-4e21-89c9-c76ee6d72ec4-istiod-ca-cert\") pod \"bac35651-e552-4e21-89c9-c76ee6d72ec4\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") "
Nov 02 09:55:59.917409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.915987    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-podinfo\") pod \"bac35651-e552-4e21-89c9-c76ee6d72ec4\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") "
Nov 02 09:55:59.917409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.916014    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-token\" (UniqueName: \"kubernetes.io/projected/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-token\") pod \"bac35651-e552-4e21-89c9-c76ee6d72ec4\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") "
Nov 02 09:55:59.917409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.916135    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-workload-certs\") pod \"bac35651-e552-4e21-89c9-c76ee6d72ec4\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") "
Nov 02 09:55:59.917409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.916171    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-data\") pod \"bac35651-e552-4e21-89c9-c76ee6d72ec4\" (UID: \"bac35651-e552-4e21-89c9-c76ee6d72ec4\") "
Nov 02 09:55:59.917409 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.917267    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac35651-e552-4e21-89c9-c76ee6d72ec4-istiod-ca-cert" pod "bac35651-e552-4e21-89c9-c76ee6d72ec4" (UID: "bac35651-e552-4e21-89c9-c76ee6d72ec4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:55:59.918128 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.917648    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-workload-socket" pod "bac35651-e552-4e21-89c9-c76ee6d72ec4" (UID: "bac35651-e552-4e21-89c9-c76ee6d72ec4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:59.918128 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.917910    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-credential-socket" pod "bac35651-e552-4e21-89c9-c76ee6d72ec4" (UID: "bac35651-e552-4e21-89c9-c76ee6d72ec4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:59.921146 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.921099    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-workload-certs" pod "bac35651-e552-4e21-89c9-c76ee6d72ec4" (UID: "bac35651-e552-4e21-89c9-c76ee6d72ec4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:55:59.982655 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:55:59.982601    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-data" pod "bac35651-e552-4e21-89c9-c76ee6d72ec4" (UID: "bac35651-e552-4e21-89c9-c76ee6d72ec4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:00.046470 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.021468    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:00.046470 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.021504    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:00.046470 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.021521    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:00.046470 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.021536    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:00.046470 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.021553    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac35651-e552-4e21-89c9-c76ee6d72ec4-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:00.111729 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.111688    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-podinfo" pod "bac35651-e552-4e21-89c9-c76ee6d72ec4" (UID: "bac35651-e552-4e21-89c9-c76ee6d72ec4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:56:00.131458 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.128464    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-token" pod "bac35651-e552-4e21-89c9-c76ee6d72ec4" (UID: "bac35651-e552-4e21-89c9-c76ee6d72ec4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:56:00.131458 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.129975    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:00.131458 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.130004    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:00.131458 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.131423    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-envoy" pod "bac35651-e552-4e21-89c9-c76ee6d72ec4" (UID: "bac35651-e552-4e21-89c9-c76ee6d72ec4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:00.142434 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.132475    4614 generic.go:356] "Generic (PLEG): container finished" podID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerID="bc0c235faea8ffbaaef6f6b3f9b140001b1e19f6db600f446f52bdde6ae4f061" exitCode=0
Nov 02 09:56:00.142434 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.133326    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"
Nov 02 09:56:00.160698 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.143098    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" event={"ID":"bac35651-e552-4e21-89c9-c76ee6d72ec4","Type":"ContainerDied","Data":"bc0c235faea8ffbaaef6f6b3f9b140001b1e19f6db600f446f52bdde6ae4f061"}
Nov 02 09:56:00.160698 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.143164    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt" event={"ID":"bac35651-e552-4e21-89c9-c76ee6d72ec4","Type":"ContainerDied","Data":"309476ebc790e44f4e160b1aad662fd3c6936f4ab663f682d9df5c033a87d113"}
Nov 02 09:56:00.160698 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.143199    4614 scope.go:122] "RemoveContainer" containerID="bc0c235faea8ffbaaef6f6b3f9b140001b1e19f6db600f446f52bdde6ae4f061"
Nov 02 09:56:00.226407 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.220613    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac35651-e552-4e21-89c9-c76ee6d72ec4-kube-api-access-5xxmb" pod "bac35651-e552-4e21-89c9-c76ee6d72ec4" (UID: "bac35651-e552-4e21-89c9-c76ee6d72ec4"). InnerVolumeSpecName "kube-api-access-5xxmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:56:00.250415 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.245377    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac35651-e552-4e21-89c9-c76ee6d72ec4-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:00.250415 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.245410    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xxmb\" (UniqueName: \"kubernetes.io/projected/bac35651-e552-4e21-89c9-c76ee6d72ec4-kube-api-access-5xxmb\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:00.304384 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.291600    4614 scope.go:122] "RemoveContainer" containerID="144bb1ac9d3051e67ca9e1d5e015a16ed4e0e120402ffe6a8892b42c84fdf5ea"
Nov 02 09:56:00.408545 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.408229    4614 scope.go:122] "RemoveContainer" containerID="bc0c235faea8ffbaaef6f6b3f9b140001b1e19f6db600f446f52bdde6ae4f061"
Nov 02 09:56:00.412412 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:56:00.409366    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0c235faea8ffbaaef6f6b3f9b140001b1e19f6db600f446f52bdde6ae4f061\": container with ID starting with bc0c235faea8ffbaaef6f6b3f9b140001b1e19f6db600f446f52bdde6ae4f061 not found: ID does not exist" containerID="bc0c235faea8ffbaaef6f6b3f9b140001b1e19f6db600f446f52bdde6ae4f061"
Nov 02 09:56:00.412412 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.409408    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0c235faea8ffbaaef6f6b3f9b140001b1e19f6db600f446f52bdde6ae4f061"} err="failed to get container status \"bc0c235faea8ffbaaef6f6b3f9b140001b1e19f6db600f446f52bdde6ae4f061\": rpc error: code = NotFound desc = could not find container \"bc0c235faea8ffbaaef6f6b3f9b140001b1e19f6db600f446f52bdde6ae4f061\": container with ID starting with bc0c235faea8ffbaaef6f6b3f9b140001b1e19f6db600f446f52bdde6ae4f061 not found: ID does not exist"
Nov 02 09:56:00.412412 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.409434    4614 scope.go:122] "RemoveContainer" containerID="144bb1ac9d3051e67ca9e1d5e015a16ed4e0e120402ffe6a8892b42c84fdf5ea"
Nov 02 09:56:00.412673 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:56:00.412483    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144bb1ac9d3051e67ca9e1d5e015a16ed4e0e120402ffe6a8892b42c84fdf5ea\": container with ID starting with 144bb1ac9d3051e67ca9e1d5e015a16ed4e0e120402ffe6a8892b42c84fdf5ea not found: ID does not exist" containerID="144bb1ac9d3051e67ca9e1d5e015a16ed4e0e120402ffe6a8892b42c84fdf5ea"
Nov 02 09:56:00.412673 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.412519    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144bb1ac9d3051e67ca9e1d5e015a16ed4e0e120402ffe6a8892b42c84fdf5ea"} err="failed to get container status \"144bb1ac9d3051e67ca9e1d5e015a16ed4e0e120402ffe6a8892b42c84fdf5ea\": rpc error: code = NotFound desc = could not find container \"144bb1ac9d3051e67ca9e1d5e015a16ed4e0e120402ffe6a8892b42c84fdf5ea\": container with ID starting with 144bb1ac9d3051e67ca9e1d5e015a16ed4e0e120402ffe6a8892b42c84fdf5ea not found: ID does not exist"
Nov 02 09:56:00.484390 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.481864    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"]
Nov 02 09:56:00.485302 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:00.485278    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a71884-openshift-default-6b7df8cf59-xxdrt"]
Nov 02 09:56:01.374378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:01.374023    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" path="/var/lib/kubelet/pods/bac35651-e552-4e21-89c9-c76ee6d72ec4/volumes"
Nov 02 09:56:03.100566 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:03.100531    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"]
Nov 02 09:56:07.696404 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:07.693482    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:08.992378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:08.974560    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"]
Nov 02 09:56:08.992378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:08.984867    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy" containerID="cri-o://70a395e4965b826ea7095eb544d67a3f752ed3769c242bd70c700f6b9fd1f994" gracePeriod=30
Nov 02 09:56:09.792575 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:56:09.792535    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:56:11.006124 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:11.006076    4614 patch_prober.go:29] interesting pod/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 09:56:11.006916 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:11.006881    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 09:56:11.391437 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:11.390445    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:12.114503 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:12.114454    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 02 09:56:12.141382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:12.119884    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:14.288366 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.288331    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:56:14.323104 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.323071    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-envoy\") pod \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") "
Nov 02 09:56:14.323104 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.323115    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-token\" (UniqueName: \"kubernetes.io/projected/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-token\") pod \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") "
Nov 02 09:56:14.323437 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.323154    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-workload-socket\") pod \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") "
Nov 02 09:56:14.323437 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.323298    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-data\") pod \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") "
Nov 02 09:56:14.323437 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.323388    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istiod-ca-cert\") pod \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") "
Nov 02 09:56:14.323569 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.323443    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-podinfo\") pod \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") "
Nov 02 09:56:14.323569 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.323488    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-credential-socket\") pod \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") "
Nov 02 09:56:14.323674 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.323565    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-workload-socket" pod "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" (UID: "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:14.323912 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.323881    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istiod-ca-cert" pod "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" (UID: "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:56:14.324109 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.324084    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-credential-socket" pod "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" (UID: "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:14.324184 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.324144    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-data" pod "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" (UID: "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:14.324247 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.324218    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:14.324307 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.324258    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:14.324307 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.324274    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:14.326814 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.326784    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-envoy" pod "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" (UID: "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:14.327153 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.327133    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-podinfo" pod "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" (UID: "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:56:14.327221 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.327195    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-token" pod "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" (UID: "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:56:14.425006 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.424976    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-kube-api-access-7cqtt\" (UniqueName: \"kubernetes.io/projected/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-kube-api-access-7cqtt\") pod \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") "
Nov 02 09:56:14.425169 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.425017    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-workload-certs\") pod \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\" (UID: \"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2\") "
Nov 02 09:56:14.425340 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.425318    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-workload-certs" pod "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" (UID: "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:14.425445 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.425428    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:14.425496 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.425448    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:14.425496 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.425459    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:14.425496 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.425467    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:14.427055 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.427032    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-kube-api-access-7cqtt" pod "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" (UID: "d5973ac7-7b39-4db6-bb4b-56014b7bf3b2"). InnerVolumeSpecName "kube-api-access-7cqtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:56:14.531449 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.528255    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cqtt\" (UniqueName: \"kubernetes.io/projected/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-kube-api-access-7cqtt\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:14.531449 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.528304    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:14.608848 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.608803    4614 generic.go:356] "Generic (PLEG): container finished" podID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerID="70a395e4965b826ea7095eb544d67a3f752ed3769c242bd70c700f6b9fd1f994" exitCode=0
Nov 02 09:56:14.608848 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.608846    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"
Nov 02 09:56:14.609115 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.608845    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" event={"ID":"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2","Type":"ContainerDied","Data":"70a395e4965b826ea7095eb544d67a3f752ed3769c242bd70c700f6b9fd1f994"}
Nov 02 09:56:14.609115 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.608906    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt" event={"ID":"d5973ac7-7b39-4db6-bb4b-56014b7bf3b2","Type":"ContainerDied","Data":"29cfce08e4f3ea99175a77051bbba7129808038a909c70573e109f2fbefae979"}
Nov 02 09:56:14.609115 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.608937    4614 scope.go:122] "RemoveContainer" containerID="70a395e4965b826ea7095eb544d67a3f752ed3769c242bd70c700f6b9fd1f994"
Nov 02 09:56:14.620229 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.620210    4614 scope.go:122] "RemoveContainer" containerID="d67c97639598c40189817f3e3d6897e3450c7f73ae1b62758a96115850cd2fad"
Nov 02 09:56:14.629513 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.629493    4614 scope.go:122] "RemoveContainer" containerID="70a395e4965b826ea7095eb544d67a3f752ed3769c242bd70c700f6b9fd1f994"
Nov 02 09:56:14.629815 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:56:14.629796    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a395e4965b826ea7095eb544d67a3f752ed3769c242bd70c700f6b9fd1f994\": container with ID starting with 70a395e4965b826ea7095eb544d67a3f752ed3769c242bd70c700f6b9fd1f994 not found: ID does not exist" containerID="70a395e4965b826ea7095eb544d67a3f752ed3769c242bd70c700f6b9fd1f994"
Nov 02 09:56:14.629866 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.629826    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a395e4965b826ea7095eb544d67a3f752ed3769c242bd70c700f6b9fd1f994"} err="failed to get container status \"70a395e4965b826ea7095eb544d67a3f752ed3769c242bd70c700f6b9fd1f994\": rpc error: code = NotFound desc = could not find container \"70a395e4965b826ea7095eb544d67a3f752ed3769c242bd70c700f6b9fd1f994\": container with ID starting with 70a395e4965b826ea7095eb544d67a3f752ed3769c242bd70c700f6b9fd1f994 not found: ID does not exist"
Nov 02 09:56:14.629866 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.629845    4614 scope.go:122] "RemoveContainer" containerID="d67c97639598c40189817f3e3d6897e3450c7f73ae1b62758a96115850cd2fad"
Nov 02 09:56:14.630085 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:56:14.630068    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67c97639598c40189817f3e3d6897e3450c7f73ae1b62758a96115850cd2fad\": container with ID starting with d67c97639598c40189817f3e3d6897e3450c7f73ae1b62758a96115850cd2fad not found: ID does not exist" containerID="d67c97639598c40189817f3e3d6897e3450c7f73ae1b62758a96115850cd2fad"
Nov 02 09:56:14.630144 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.630088    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67c97639598c40189817f3e3d6897e3450c7f73ae1b62758a96115850cd2fad"} err="failed to get container status \"d67c97639598c40189817f3e3d6897e3450c7f73ae1b62758a96115850cd2fad\": rpc error: code = NotFound desc = could not find container \"d67c97639598c40189817f3e3d6897e3450c7f73ae1b62758a96115850cd2fad\": container with ID starting with d67c97639598c40189817f3e3d6897e3450c7f73ae1b62758a96115850cd2fad not found: ID does not exist"
Nov 02 09:56:14.656081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.656056    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"]
Nov 02 09:56:14.661020 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:14.660994    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ae13d4-openshift-default-74cd94fc9f-mvzmt"]
Nov 02 09:56:15.316969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:15.316894    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" path="/var/lib/kubelet/pods/d5973ac7-7b39-4db6-bb4b-56014b7bf3b2/volumes"
Nov 02 09:56:18.122785 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.122749    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb" podUID="75f06396-128b-4121-af72-94df8d58bc59" containerName="main" containerID="cri-o://4780149ee7bc98533a40ab19cfd9a7147ae7974e0844a7cb293e27d70bd39303" gracePeriod=45
Nov 02 09:56:18.413247 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.413223    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:56:18.535372 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.531459    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-home\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-home\") pod \"75f06396-128b-4121-af72-94df8d58bc59\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") "
Nov 02 09:56:18.535372 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.531562    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-tmp-dir\") pod \"75f06396-128b-4121-af72-94df8d58bc59\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") "
Nov 02 09:56:18.535372 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.531818    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f06396-128b-4121-af72-94df8d58bc59-kube-api-access-4vqcc\" (UniqueName: \"kubernetes.io/projected/75f06396-128b-4121-af72-94df8d58bc59-kube-api-access-4vqcc\") pod \"75f06396-128b-4121-af72-94df8d58bc59\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") "
Nov 02 09:56:18.535372 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.531910    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-dshm\") pod \"75f06396-128b-4121-af72-94df8d58bc59\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") "
Nov 02 09:56:18.535372 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.531948    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-model-cache\") pod \"75f06396-128b-4121-af72-94df8d58bc59\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") "
Nov 02 09:56:18.535372 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.532016    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f06396-128b-4121-af72-94df8d58bc59-tls-certs\" (UniqueName: \"kubernetes.io/secret/75f06396-128b-4121-af72-94df8d58bc59-tls-certs\") pod \"75f06396-128b-4121-af72-94df8d58bc59\" (UID: \"75f06396-128b-4121-af72-94df8d58bc59\") "
Nov 02 09:56:18.553373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.552321    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-home" pod "75f06396-128b-4121-af72-94df8d58bc59" (UID: "75f06396-128b-4121-af72-94df8d58bc59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:18.553373 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.552711    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-tmp-dir" pod "75f06396-128b-4121-af72-94df8d58bc59" (UID: "75f06396-128b-4121-af72-94df8d58bc59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:18.564386 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.556927    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-model-cache" pod "75f06396-128b-4121-af72-94df8d58bc59" (UID: "75f06396-128b-4121-af72-94df8d58bc59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:18.577270 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.577193    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f06396-128b-4121-af72-94df8d58bc59-tls-certs" pod "75f06396-128b-4121-af72-94df8d58bc59" (UID: "75f06396-128b-4121-af72-94df8d58bc59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:56:18.579742 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.579251    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-dshm" pod "75f06396-128b-4121-af72-94df8d58bc59" (UID: "75f06396-128b-4121-af72-94df8d58bc59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:56:18.586667 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.586555    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f06396-128b-4121-af72-94df8d58bc59-kube-api-access-4vqcc" pod "75f06396-128b-4121-af72-94df8d58bc59" (UID: "75f06396-128b-4121-af72-94df8d58bc59"). InnerVolumeSpecName "kube-api-access-4vqcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:56:18.634471 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.634395    4614 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-home\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:18.634471 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.634430    4614 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-tmp-dir\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:18.634471 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.634444    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vqcc\" (UniqueName: \"kubernetes.io/projected/75f06396-128b-4121-af72-94df8d58bc59-kube-api-access-4vqcc\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:18.634471 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.634457    4614 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-dshm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:18.634471 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.634471    4614 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75f06396-128b-4121-af72-94df8d58bc59-model-cache\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:18.634708 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.634484    4614 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75f06396-128b-4121-af72-94df8d58bc59-tls-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:18.638134 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.638104    4614 generic.go:356] "Generic (PLEG): container finished" podID="75f06396-128b-4121-af72-94df8d58bc59" containerID="4780149ee7bc98533a40ab19cfd9a7147ae7974e0844a7cb293e27d70bd39303" exitCode=0
Nov 02 09:56:18.638234 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.638152    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"
Nov 02 09:56:18.638234 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.638160    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb" event={"ID":"75f06396-128b-4121-af72-94df8d58bc59","Type":"ContainerDied","Data":"4780149ee7bc98533a40ab19cfd9a7147ae7974e0844a7cb293e27d70bd39303"}
Nov 02 09:56:18.638234 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.638197    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb" event={"ID":"75f06396-128b-4121-af72-94df8d58bc59","Type":"ContainerDied","Data":"a3cf2785adc50c286fdb6e6b8f2a6b23b0485ffc93a606bcceed3fde051a5a35"}
Nov 02 09:56:18.638234 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.638219    4614 scope.go:122] "RemoveContainer" containerID="4780149ee7bc98533a40ab19cfd9a7147ae7974e0844a7cb293e27d70bd39303"
Nov 02 09:56:18.648806 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.648788    4614 scope.go:122] "RemoveContainer" containerID="4780149ee7bc98533a40ab19cfd9a7147ae7974e0844a7cb293e27d70bd39303"
Nov 02 09:56:18.649059 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:56:18.649039    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4780149ee7bc98533a40ab19cfd9a7147ae7974e0844a7cb293e27d70bd39303\": container with ID starting with 4780149ee7bc98533a40ab19cfd9a7147ae7974e0844a7cb293e27d70bd39303 not found: ID does not exist" containerID="4780149ee7bc98533a40ab19cfd9a7147ae7974e0844a7cb293e27d70bd39303"
Nov 02 09:56:18.649109 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.649068    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4780149ee7bc98533a40ab19cfd9a7147ae7974e0844a7cb293e27d70bd39303"} err="failed to get container status \"4780149ee7bc98533a40ab19cfd9a7147ae7974e0844a7cb293e27d70bd39303\": rpc error: code = NotFound desc = could not find container \"4780149ee7bc98533a40ab19cfd9a7147ae7974e0844a7cb293e27d70bd39303\": container with ID starting with 4780149ee7bc98533a40ab19cfd9a7147ae7974e0844a7cb293e27d70bd39303 not found: ID does not exist"
Nov 02 09:56:18.664479 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.664454    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"]
Nov 02 09:56:18.667901 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:18.667881    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-9a69d1/explicit-model-9a69d1-kserve-79bc8b8c4c-76mhb"]
Nov 02 09:56:19.317305 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:19.317271    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f06396-128b-4121-af72-94df8d58bc59" path="/var/lib/kubelet/pods/75f06396-128b-4121-af72-94df8d58bc59/volumes"
Nov 02 09:56:20.582444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:20.582412    4614 scope.go:122] "RemoveContainer" containerID="019fcd0a589e89f51219810d9f03686919fd8973b1a7b02e60ac0d3493ac0539"
Nov 02 09:56:20.592939 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:20.592812    4614 scope.go:122] "RemoveContainer" containerID="6f952144c45b318ef83ea3d976a5d2230cddb77ff4b81a9a19510cfd4479a6b3"
Nov 02 09:56:20.622638 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:20.603371    4614 scope.go:122] "RemoveContainer" containerID="d5938b107644a6149c9e517c9e6237a5882daf90858d4aeee0992bec09de49b7"
Nov 02 09:56:39.636380 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:56:39.636327    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:56:45.867964 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:45.867924    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"]
Nov 02 09:56:45.882312 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:45.882271    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"]
Nov 02 09:56:50.869470 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:50.869421    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c" podUID="342c117c-0a54-434c-8a32-8f49498176ab" containerName="payload-processing" containerID="cri-o://9dc3e970d06a37e4286940832119b2c401116c42da6afa5da2efdf9bf1f26ba2" gracePeriod=25
Nov 02 09:56:50.885763 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:50.885730    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9" podUID="b0089d27-a642-4ce9-8e37-073721510df4" containerName="payload-pre-processing" containerID="cri-o://76c4c0fa3342e6f17cda185b885e1ed8734b36132a436f7758449c559d957dfa" gracePeriod=25
Nov 02 09:56:50.925659 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:50.924441    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:50.966380 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:56:50.965498    4614 logging.go:55] [core] [Channel #556 SubChannel #557]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.117:50052", ServerName: "10.132.0.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.117:50052: connect: connection refused"
Nov 02 09:56:51.064845 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:51.064787    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:51.260377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:51.259532    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:56:51.260377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:51.259623    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:51.260377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:51.260098    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:51.456368 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:51.456254    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:51.777809 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:51.777766    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 02 09:56:51.783603 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:51.783544    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:51.930891 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:51.930834    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:56:51.948384 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:51.946638    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:56:52.165066 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:56:51.980676    4614 logging.go:55] [core] [Channel #559 SubChannel #561]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.116:50052", ServerName: "10.132.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.116:50052: connect: connection refused"
Nov 02 09:56:52.202449 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:51.996870    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c" podUID="342c117c-0a54-434c-8a32-8f49498176ab" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.117:50052\" within 1s: context deadline exceeded"
Nov 02 09:56:52.202449 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:56:51.996963    4614 logging.go:55] [core] [Channel #556 SubChannel #557]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.117:50052", ServerName: "10.132.0.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.117:50052: operation was canceled"
Nov 02 09:56:52.202449 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.163055    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 09:56:52.305674 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.305468    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"
Nov 02 09:56:52.307648 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.307579    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"
Nov 02 09:56:52.341622 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.339252    4614 generic.go:356] "Generic (PLEG): container finished" podID="b0089d27-a642-4ce9-8e37-073721510df4" containerID="76c4c0fa3342e6f17cda185b885e1ed8734b36132a436f7758449c559d957dfa" exitCode=0
Nov 02 09:56:52.341622 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.339657    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9" event={"ID":"b0089d27-a642-4ce9-8e37-073721510df4","Type":"ContainerDied","Data":"76c4c0fa3342e6f17cda185b885e1ed8734b36132a436f7758449c559d957dfa"}
Nov 02 09:56:52.341622 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.339717    4614 scope.go:122] "RemoveContainer" containerID="76c4c0fa3342e6f17cda185b885e1ed8734b36132a436f7758449c559d957dfa"
Nov 02 09:56:52.341622 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.339857    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"
Nov 02 09:56:52.371889 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.368593    4614 generic.go:356] "Generic (PLEG): container finished" podID="342c117c-0a54-434c-8a32-8f49498176ab" containerID="9dc3e970d06a37e4286940832119b2c401116c42da6afa5da2efdf9bf1f26ba2" exitCode=0
Nov 02 09:56:52.371889 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.368822    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c" event={"ID":"342c117c-0a54-434c-8a32-8f49498176ab","Type":"ContainerDied","Data":"9dc3e970d06a37e4286940832119b2c401116c42da6afa5da2efdf9bf1f26ba2"}
Nov 02 09:56:52.371889 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.368890    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c" event={"ID":"342c117c-0a54-434c-8a32-8f49498176ab","Type":"ContainerDied","Data":"5cff45e4a00314adb407556eb12e641080c29d655ddd6fa9ad2a04b04c84ed8b"}
Nov 02 09:56:52.371889 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.369035    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"
Nov 02 09:56:52.409828 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.409794    4614 scope.go:122] "RemoveContainer" containerID="9dc3e970d06a37e4286940832119b2c401116c42da6afa5da2efdf9bf1f26ba2"
Nov 02 09:56:52.426627 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.426288    4614 scope.go:122] "RemoveContainer" containerID="9dc3e970d06a37e4286940832119b2c401116c42da6afa5da2efdf9bf1f26ba2"
Nov 02 09:56:52.427192 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:56:52.427079    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc3e970d06a37e4286940832119b2c401116c42da6afa5da2efdf9bf1f26ba2\": container with ID starting with 9dc3e970d06a37e4286940832119b2c401116c42da6afa5da2efdf9bf1f26ba2 not found: ID does not exist" containerID="9dc3e970d06a37e4286940832119b2c401116c42da6afa5da2efdf9bf1f26ba2"
Nov 02 09:56:52.427192 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.427121    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc3e970d06a37e4286940832119b2c401116c42da6afa5da2efdf9bf1f26ba2"} err="failed to get container status \"9dc3e970d06a37e4286940832119b2c401116c42da6afa5da2efdf9bf1f26ba2\": rpc error: code = NotFound desc = could not find container \"9dc3e970d06a37e4286940832119b2c401116c42da6afa5da2efdf9bf1f26ba2\": container with ID starting with 9dc3e970d06a37e4286940832119b2c401116c42da6afa5da2efdf9bf1f26ba2 not found: ID does not exist"
Nov 02 09:56:52.473907 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.473879    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342c117c-0a54-434c-8a32-8f49498176ab-kube-api-access-5kcgs\" (UniqueName: \"kubernetes.io/projected/342c117c-0a54-434c-8a32-8f49498176ab-kube-api-access-5kcgs\") pod \"342c117c-0a54-434c-8a32-8f49498176ab\" (UID: \"342c117c-0a54-434c-8a32-8f49498176ab\") "
Nov 02 09:56:52.474186 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.473986    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/342c117c-0a54-434c-8a32-8f49498176ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342c117c-0a54-434c-8a32-8f49498176ab-plugins-config-volume\") pod \"342c117c-0a54-434c-8a32-8f49498176ab\" (UID: \"342c117c-0a54-434c-8a32-8f49498176ab\") "
Nov 02 09:56:52.474186 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.474030    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0089d27-a642-4ce9-8e37-073721510df4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0089d27-a642-4ce9-8e37-073721510df4-plugins-config-volume\") pod \"b0089d27-a642-4ce9-8e37-073721510df4\" (UID: \"b0089d27-a642-4ce9-8e37-073721510df4\") "
Nov 02 09:56:52.474186 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.474054    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0089d27-a642-4ce9-8e37-073721510df4-kube-api-access-lnp5m\" (UniqueName: \"kubernetes.io/projected/b0089d27-a642-4ce9-8e37-073721510df4-kube-api-access-lnp5m\") pod \"b0089d27-a642-4ce9-8e37-073721510df4\" (UID: \"b0089d27-a642-4ce9-8e37-073721510df4\") "
Nov 02 09:56:52.475680 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.474603    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342c117c-0a54-434c-8a32-8f49498176ab-plugins-config-volume" pod "342c117c-0a54-434c-8a32-8f49498176ab" (UID: "342c117c-0a54-434c-8a32-8f49498176ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:56:52.475680 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.475491    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/342c117c-0a54-434c-8a32-8f49498176ab-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:52.475852 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.475814    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0089d27-a642-4ce9-8e37-073721510df4-plugins-config-volume" pod "b0089d27-a642-4ce9-8e37-073721510df4" (UID: "b0089d27-a642-4ce9-8e37-073721510df4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:56:52.477065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.477045    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342c117c-0a54-434c-8a32-8f49498176ab-kube-api-access-5kcgs" pod "342c117c-0a54-434c-8a32-8f49498176ab" (UID: "342c117c-0a54-434c-8a32-8f49498176ab"). InnerVolumeSpecName "kube-api-access-5kcgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:56:52.477780 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.477762    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0089d27-a642-4ce9-8e37-073721510df4-kube-api-access-lnp5m" pod "b0089d27-a642-4ce9-8e37-073721510df4" (UID: "b0089d27-a642-4ce9-8e37-073721510df4"). InnerVolumeSpecName "kube-api-access-lnp5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:56:52.619372 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.618188    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kcgs\" (UniqueName: \"kubernetes.io/projected/342c117c-0a54-434c-8a32-8f49498176ab-kube-api-access-5kcgs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:52.619372 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.618225    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0089d27-a642-4ce9-8e37-073721510df4-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:52.619372 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.618241    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnp5m\" (UniqueName: \"kubernetes.io/projected/b0089d27-a642-4ce9-8e37-073721510df4-kube-api-access-lnp5m\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:52.684861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.684828    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"]
Nov 02 09:56:52.689247 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.689215    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9"]
Nov 02 09:56:52.701896 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.701865    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"]
Nov 02 09:56:52.705124 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.705089    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9a69d1-cbbb9c4d9-kp42c"]
Nov 02 09:56:52.997090 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:52.992154    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9a69d1-7869845747-mnbt9" podUID="b0089d27-a642-4ce9-8e37-073721510df4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.116:50052\" within 1s: context deadline exceeded"
Nov 02 09:56:53.322783 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:53.322703    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="342c117c-0a54-434c-8a32-8f49498176ab" path="/var/lib/kubelet/pods/342c117c-0a54-434c-8a32-8f49498176ab/volumes"
Nov 02 09:56:53.323502 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:53.323484    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0089d27-a642-4ce9-8e37-073721510df4" path="/var/lib/kubelet/pods/b0089d27-a642-4ce9-8e37-073721510df4/volumes"
Nov 02 09:56:53.436460 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:53.431089    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"]
Nov 02 09:56:53.436460 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:53.431346    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn" podUID="29e4deca-3ded-4b46-983c-911a324f8eaa" containerName="maas-api" containerID="cri-o://31f6ede88deaca2e41bf0fb77d0ff2314317f954b4b3b16b44176c12fc732087" gracePeriod=30
Nov 02 09:56:53.960269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:53.960238    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:56:53.969676 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:53.969648    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29e4deca-3ded-4b46-983c-911a324f8eaa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e4deca-3ded-4b46-983c-911a324f8eaa-metrics-certs\") pod \"29e4deca-3ded-4b46-983c-911a324f8eaa\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") "
Nov 02 09:56:53.969814 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:53.969686    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29e4deca-3ded-4b46-983c-911a324f8eaa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e4deca-3ded-4b46-983c-911a324f8eaa-maas-api-tls\") pod \"29e4deca-3ded-4b46-983c-911a324f8eaa\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") "
Nov 02 09:56:53.969814 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:53.969794    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e4deca-3ded-4b46-983c-911a324f8eaa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e4deca-3ded-4b46-983c-911a324f8eaa-openshift-service-ca\") pod \"29e4deca-3ded-4b46-983c-911a324f8eaa\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") "
Nov 02 09:56:53.969902 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:53.969887    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e4deca-3ded-4b46-983c-911a324f8eaa-kube-api-access-nbjnt\" (UniqueName: \"kubernetes.io/projected/29e4deca-3ded-4b46-983c-911a324f8eaa-kube-api-access-nbjnt\") pod \"29e4deca-3ded-4b46-983c-911a324f8eaa\" (UID: \"29e4deca-3ded-4b46-983c-911a324f8eaa\") "
Nov 02 09:56:53.970142 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:53.970116    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e4deca-3ded-4b46-983c-911a324f8eaa-openshift-service-ca" pod "29e4deca-3ded-4b46-983c-911a324f8eaa" (UID: "29e4deca-3ded-4b46-983c-911a324f8eaa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:56:53.970493 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:53.970474    4614 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e4deca-3ded-4b46-983c-911a324f8eaa-openshift-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:53.971928 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:53.971900    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e4deca-3ded-4b46-983c-911a324f8eaa-kube-api-access-nbjnt" pod "29e4deca-3ded-4b46-983c-911a324f8eaa" (UID: "29e4deca-3ded-4b46-983c-911a324f8eaa"). InnerVolumeSpecName "kube-api-access-nbjnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:56:53.971928 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:53.971907    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e4deca-3ded-4b46-983c-911a324f8eaa-metrics-certs" pod "29e4deca-3ded-4b46-983c-911a324f8eaa" (UID: "29e4deca-3ded-4b46-983c-911a324f8eaa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:56:53.972065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:53.971940    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e4deca-3ded-4b46-983c-911a324f8eaa-maas-api-tls" pod "29e4deca-3ded-4b46-983c-911a324f8eaa" (UID: "29e4deca-3ded-4b46-983c-911a324f8eaa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:56:54.071566 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:54.071528    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbjnt\" (UniqueName: \"kubernetes.io/projected/29e4deca-3ded-4b46-983c-911a324f8eaa-kube-api-access-nbjnt\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:54.071566 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:54.071561    4614 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29e4deca-3ded-4b46-983c-911a324f8eaa-metrics-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:54.071566 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:54.071575    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e4deca-3ded-4b46-983c-911a324f8eaa-maas-api-tls\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:56:54.385203 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:54.385169    4614 generic.go:356] "Generic (PLEG): container finished" podID="29e4deca-3ded-4b46-983c-911a324f8eaa" containerID="31f6ede88deaca2e41bf0fb77d0ff2314317f954b4b3b16b44176c12fc732087" exitCode=0
Nov 02 09:56:54.385417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:54.385215    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"
Nov 02 09:56:54.385417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:54.385243    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn" event={"ID":"29e4deca-3ded-4b46-983c-911a324f8eaa","Type":"ContainerDied","Data":"31f6ede88deaca2e41bf0fb77d0ff2314317f954b4b3b16b44176c12fc732087"}
Nov 02 09:56:54.385417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:54.385281    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn" event={"ID":"29e4deca-3ded-4b46-983c-911a324f8eaa","Type":"ContainerDied","Data":"5c9e18ff5ab1756a23047f47c73f5418cc6c3a5fa769927c012d2c9ea332e1b6"}
Nov 02 09:56:54.385417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:54.385303    4614 scope.go:122] "RemoveContainer" containerID="31f6ede88deaca2e41bf0fb77d0ff2314317f954b4b3b16b44176c12fc732087"
Nov 02 09:56:54.396109 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:54.396090    4614 scope.go:122] "RemoveContainer" containerID="31f6ede88deaca2e41bf0fb77d0ff2314317f954b4b3b16b44176c12fc732087"
Nov 02 09:56:54.396414 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:56:54.396394    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f6ede88deaca2e41bf0fb77d0ff2314317f954b4b3b16b44176c12fc732087\": container with ID starting with 31f6ede88deaca2e41bf0fb77d0ff2314317f954b4b3b16b44176c12fc732087 not found: ID does not exist" containerID="31f6ede88deaca2e41bf0fb77d0ff2314317f954b4b3b16b44176c12fc732087"
Nov 02 09:56:54.396486 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:54.396421    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f6ede88deaca2e41bf0fb77d0ff2314317f954b4b3b16b44176c12fc732087"} err="failed to get container status \"31f6ede88deaca2e41bf0fb77d0ff2314317f954b4b3b16b44176c12fc732087\": rpc error: code = NotFound desc = could not find container \"31f6ede88deaca2e41bf0fb77d0ff2314317f954b4b3b16b44176c12fc732087\": container with ID starting with 31f6ede88deaca2e41bf0fb77d0ff2314317f954b4b3b16b44176c12fc732087 not found: ID does not exist"
Nov 02 09:56:54.413574 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:54.413552    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"]
Nov 02 09:56:54.420213 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:54.420192    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-9a69d1-8559b45b79-6s5fn"]
Nov 02 09:56:55.318376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:56:55.318318    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e4deca-3ded-4b46-983c-911a324f8eaa" path="/var/lib/kubelet/pods/29e4deca-3ded-4b46-983c-911a324f8eaa/volumes"
Nov 02 09:57:01.969424 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:01.968749    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:57:02.279452 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:02.262444    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:57:02.367120 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:02.367076    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Nov 02 09:57:03.022473 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.022443    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"]
Nov 02 09:57:03.023953 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.023927    4614 cpu_manager.go:414] "containerMap: removing container" podUID="75f06396-128b-4121-af72-94df8d58bc59" containerName="main"
Nov 02 09:57:03.024104 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.024091    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f06396-128b-4121-af72-94df8d58bc59" containerName="main"
Nov 02 09:57:03.024219 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.024208    4614 cpu_manager.go:414] "containerMap: removing container" podUID="342c117c-0a54-434c-8a32-8f49498176ab" containerName="payload-processing"
Nov 02 09:57:03.024295 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.024285    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342c117c-0a54-434c-8a32-8f49498176ab" containerName="payload-processing"
Nov 02 09:57:03.024411 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.024399    4614 cpu_manager.go:414] "containerMap: removing container" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerName="istio-proxy"
Nov 02 09:57:03.024513 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.024501    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerName="istio-proxy"
Nov 02 09:57:03.024588 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.024578    4614 cpu_manager.go:414] "containerMap: removing container" podUID="29e4deca-3ded-4b46-983c-911a324f8eaa" containerName="maas-api"
Nov 02 09:57:03.024668 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.024658    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e4deca-3ded-4b46-983c-911a324f8eaa" containerName="maas-api"
Nov 02 09:57:03.025309 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.025292    4614 cpu_manager.go:414] "containerMap: removing container" podUID="b0089d27-a642-4ce9-8e37-073721510df4" containerName="payload-pre-processing"
Nov 02 09:57:03.025457 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.025442    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0089d27-a642-4ce9-8e37-073721510df4" containerName="payload-pre-processing"
Nov 02 09:57:03.025582 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.025569    4614 cpu_manager.go:414] "containerMap: removing container" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerName="istio-proxy"
Nov 02 09:57:03.025697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.025684    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerName="istio-proxy"
Nov 02 09:57:03.025796 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.025787    4614 cpu_manager.go:414] "containerMap: removing container" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy"
Nov 02 09:57:03.025866 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.025856    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy"
Nov 02 09:57:03.025937 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.025928    4614 cpu_manager.go:414] "containerMap: removing container" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy"
Nov 02 09:57:03.026006 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.025996    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy"
Nov 02 09:57:03.026239 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.026223    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy"
Nov 02 09:57:03.026465 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.026337    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5973ac7-7b39-4db6-bb4b-56014b7bf3b2" containerName="istio-proxy"
Nov 02 09:57:03.026606 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.026592    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f06396-128b-4121-af72-94df8d58bc59" containerName="main"
Nov 02 09:57:03.026697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.026686    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="342c117c-0a54-434c-8a32-8f49498176ab" containerName="payload-processing"
Nov 02 09:57:03.026782 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.026771    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerName="istio-proxy"
Nov 02 09:57:03.026870 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.026861    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e4deca-3ded-4b46-983c-911a324f8eaa" containerName="maas-api"
Nov 02 09:57:03.026944 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.026936    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0089d27-a642-4ce9-8e37-073721510df4" containerName="payload-pre-processing"
Nov 02 09:57:03.034569 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.034547    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"]
Nov 02 09:57:03.034838 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.034774    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.054515 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.054447    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7btxg\" (UniqueName: \"kubernetes.io/projected/f58af013-7d2b-4a6b-a55f-e056148cd172-kube-api-access-7btxg\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.054663 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.054547    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-data\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.054663 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.054595    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-workload-socket\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.054663 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.054659    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-token\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.054838 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.054695    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-envoy\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.054838 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.054771    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58af013-7d2b-4a6b-a55f-e056148cd172-istiod-ca-cert\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.054838 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.054831    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-workload-certs\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.054998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.054861    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-credential-socket\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.054998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.054928    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-podinfo\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.126445 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.125476    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-0c5f4e-openshift-default-dockercfg-dxvfc\""
Nov 02 09:57:03.161998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.161951    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58af013-7d2b-4a6b-a55f-e056148cd172-istiod-ca-cert\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.238635 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.238599    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-workload-certs\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.239313 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.239290    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-credential-socket\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.239931 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.239913    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-podinfo\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.240460 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.240440    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7btxg\" (UniqueName: \"kubernetes.io/projected/f58af013-7d2b-4a6b-a55f-e056148cd172-kube-api-access-7btxg\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.244480 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.239802    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-credential-socket\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.244619 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.163082    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58af013-7d2b-4a6b-a55f-e056148cd172-istiod-ca-cert\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.244706 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.239209    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-workload-certs\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.244781 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.238445    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"]
Nov 02 09:57:03.271375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.245838    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac35651-e552-4e21-89c9-c76ee6d72ec4" containerName="istio-proxy"
Nov 02 09:57:03.296842 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.277067    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.297222 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.297188    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-data\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.304376 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.297442    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-workload-socket\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.304675 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.304647    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-token\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.304850 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.304833    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-envoy\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.307819 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.301346    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-629129-openshift-default-dockercfg-mggz7\""
Nov 02 09:57:03.307819 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.301894    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-data\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.307819 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.302763    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-workload-socket\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.327802 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.321717    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7btxg\" (UniqueName: \"kubernetes.io/projected/f58af013-7d2b-4a6b-a55f-e056148cd172-kube-api-access-7btxg\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.331675 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.331615    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-token\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.416991 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.416951    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.417241 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.417220    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.429043 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.428949    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.429670 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.429638    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.490430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.485712    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.490430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.485776    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74kc2\" (UniqueName: \"kubernetes.io/projected/4c42817a-b1ca-486d-8f46-3db3bbf9346c-kube-api-access-74kc2\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.490430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.485884    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.490430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.486043    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.490430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.486106    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.490430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.473915    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-podinfo\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.490430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.490106    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"]
Nov 02 09:57:03.506403 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.493264    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-envoy\") pod \"e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") " pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.598784 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.597970    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.598784 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.598025    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.598784 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.598061    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74kc2\" (UniqueName: \"kubernetes.io/projected/4c42817a-b1ca-486d-8f46-3db3bbf9346c-kube-api-access-74kc2\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.598784 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.598154    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.598784 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.598223    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.598784 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.598266    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.598784 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.598345    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.598784 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.598385    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.598784 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.598442    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.599379 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.599273    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.601610 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.599905    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.601610 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.600643    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.601610 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.600992    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.601610 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.601387    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.610266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.608798    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.611961 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.611913    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.613503 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.613481    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.615892 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.615848    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74kc2\" (UniqueName: \"kubernetes.io/projected/4c42817a-b1ca-486d-8f46-3db3bbf9346c-kube-api-access-74kc2\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.656938 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.656905    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:03.831709 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.831677    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:03.956693 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:57:03.956659    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58af013_7d2b_4a6b_a55f_e056148cd172.slice/crio-5113edb945359847e7c93837f4eaf747a8c1e1728ed4282224582371ab86ad00 WatchSource:0}: Error finding container 5113edb945359847e7c93837f4eaf747a8c1e1728ed4282224582371ab86ad00: Status 404 returned error can't find the container with id 5113edb945359847e7c93837f4eaf747a8c1e1728ed4282224582371ab86ad00
Nov 02 09:57:03.957455 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.957422    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"]
Nov 02 09:57:03.959374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.959328    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:57:03.959484 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.959418    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:57:03.959484 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:03.959464    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:57:04.120340 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:57:04.120304    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c42817a_b1ca_486d_8f46_3db3bbf9346c.slice/crio-463c062b720a2e91e84ddf25f829235714239a99dcda7e42ca686dce5918aec5 WatchSource:0}: Error finding container 463c062b720a2e91e84ddf25f829235714239a99dcda7e42ca686dce5918aec5: Status 404 returned error can't find the container with id 463c062b720a2e91e84ddf25f829235714239a99dcda7e42ca686dce5918aec5
Nov 02 09:57:04.120707 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:04.120429    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"]
Nov 02 09:57:04.122888 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:04.122853    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:57:04.122992 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:04.122919    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:57:04.122992 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:04.122943    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 09:57:04.619489 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:04.619455    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg" event={"ID":"f58af013-7d2b-4a6b-a55f-e056148cd172","Type":"ContainerStarted","Data":"0db784a64c1860f7c4c3fca066c3736bb98f3d322ef68790c87da7af98a05d98"}
Nov 02 09:57:04.619655 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:04.619509    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg" event={"ID":"f58af013-7d2b-4a6b-a55f-e056148cd172","Type":"ContainerStarted","Data":"5113edb945359847e7c93837f4eaf747a8c1e1728ed4282224582371ab86ad00"}
Nov 02 09:57:04.621324 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:04.621282    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn" event={"ID":"4c42817a-b1ca-486d-8f46-3db3bbf9346c","Type":"ContainerStarted","Data":"06f7f1fa25c65eead796e403001245f5692e34630823f7ed59a085f14bc3ec0a"}
Nov 02 09:57:04.621498 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:04.621338    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn" event={"ID":"4c42817a-b1ca-486d-8f46-3db3bbf9346c","Type":"ContainerStarted","Data":"463c062b720a2e91e84ddf25f829235714239a99dcda7e42ca686dce5918aec5"}
Nov 02 09:57:04.649318 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:04.646475    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg" podStartSLOduration=2.646453614 podStartE2EDuration="2.646453614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:57:04.643459096 +0000 UTC m=+1305.476117177" watchObservedRunningTime="2026-11-02 09:57:04.646453614 +0000 UTC m=+1305.479111690"
Nov 02 09:57:04.672472 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:04.672421    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:04.694276 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:04.694216    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn" podStartSLOduration=1.6942014520000002 podStartE2EDuration="1.694201452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:57:04.691739978 +0000 UTC m=+1305.524398059" watchObservedRunningTime="2026-11-02 09:57:04.694201452 +0000 UTC m=+1305.526859531"
Nov 02 09:57:04.843380 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:04.840569    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:05.663141 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:05.663107    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:05.844377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:05.842455    4614 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 09:57:05.844377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:05.842518    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn" podUID="4c42817a-b1ca-486d-8f46-3db3bbf9346c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 09:57:06.652377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:06.650639    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:06.656570 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:06.656543    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:06.843378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:06.835717    4614 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 09:57:06.843378 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:06.835778    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn" podUID="4c42817a-b1ca-486d-8f46-3db3bbf9346c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 09:57:07.324399 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.321491    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"]
Nov 02 09:57:07.376696 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.373515    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.378438 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.378418    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-0c5f4e\"/\"kube-root-ca.crt\""
Nov 02 09:57:07.378697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.378685    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-0c5f4e\"/\"openshift-service-ca.crt\""
Nov 02 09:57:07.379501 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.379483    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"]
Nov 02 09:57:07.381524 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.381510    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-0c5f4e\"/\"default-dockercfg-tvr7w\""
Nov 02 09:57:07.381744 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.381729    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-0c5f4e\"/\"noten-model-0c5f4e-kserve-self-signed-certs\""
Nov 02 09:57:07.416880 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.416848    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-dshm\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.416880 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.416880    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-tls-certs\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.417099 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.417054    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-home\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.417178 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.417158    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-tmp-dir\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.417283 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.417270    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-model-cache\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.417382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.417302    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfxdb\" (UniqueName: \"kubernetes.io/projected/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-kube-api-access-rfxdb\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.520304 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.520260    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-dshm\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.520638 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.520614    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-tls-certs\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.520935 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.520915    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-home\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.521109 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.521092    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-tmp-dir\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.521272 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.521257    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-model-cache\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.521391 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.521375    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfxdb\" (UniqueName: \"kubernetes.io/projected/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-kube-api-access-rfxdb\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.528789 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.528759    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-home\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.529655 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.529632    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-tmp-dir\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.538382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.530083    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-model-cache\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.539300 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.538754    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-dshm\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.556076 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.556042    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfxdb\" (UniqueName: \"kubernetes.io/projected/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-kube-api-access-rfxdb\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.582096 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.582019    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-tls-certs\") pod \"noten-model-0c5f4e-kserve-7b495865d9-8rf46\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") " pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.714377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.714211    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:07.839428 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.832042    4614 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 09:57:07.839428 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:07.832115    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn" podUID="4c42817a-b1ca-486d-8f46-3db3bbf9346c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 09:57:08.104762 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:08.104731    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"]
Nov 02 09:57:08.105752 ip-10-0-134-174 kubenswrapper[4614]: W1102 09:57:08.105722    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4c19aac_3aca_42e1_bd87_22c2a3a3509b.slice/crio-445d0abff84be2463b0b1e654819dacf2d2517338a9671e3ea861d66fe30df8b WatchSource:0}: Error finding container 445d0abff84be2463b0b1e654819dacf2d2517338a9671e3ea861d66fe30df8b: Status 404 returned error can't find the container with id 445d0abff84be2463b0b1e654819dacf2d2517338a9671e3ea861d66fe30df8b
Nov 02 09:57:08.721027 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:08.720991    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46" event={"ID":"b4c19aac-3aca-42e1-bd87-22c2a3a3509b","Type":"ContainerStarted","Data":"434e049f9d264193e21e4c7d7a657db360403bb70b9c1f2c1e1709e9b7889908"}
Nov 02 09:57:08.721276 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:08.721254    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46" event={"ID":"b4c19aac-3aca-42e1-bd87-22c2a3a3509b","Type":"ContainerStarted","Data":"445d0abff84be2463b0b1e654819dacf2d2517338a9671e3ea861d66fe30df8b"}
Nov 02 09:57:08.757991 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:08.757937    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46" podStartSLOduration=1.757920676 podStartE2EDuration="1.757920676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 09:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 09:57:08.747591537 +0000 UTC m=+1309.580249617" watchObservedRunningTime="2026-11-02 09:57:08.757920676 +0000 UTC m=+1309.590578756"
Nov 02 09:57:08.842744 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:08.842705    4614 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 09:57:08.843003 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:08.842976    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn" podUID="4c42817a-b1ca-486d-8f46-3db3bbf9346c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 09:57:09.668317 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:57:09.668273    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:57:09.716979 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:09.716941    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:09.850383 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:09.849779    4614 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 09:57:09.850383 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:09.849839    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn" podUID="4c42817a-b1ca-486d-8f46-3db3bbf9346c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 09:57:11.287248 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:11.287220    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:11.287811 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:11.287784    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:11.288952 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:11.288928    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 09:57:13.547517 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:13.547476    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"]
Nov 02 09:57:13.726384 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:13.716288    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"]
Nov 02 09:57:13.742375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:13.738636    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg" podUID="f58af013-7d2b-4a6b-a55f-e056148cd172" containerName="istio-proxy" containerID="cri-o://0db784a64c1860f7c4c3fca066c3736bb98f3d322ef68790c87da7af98a05d98" gracePeriod=30
Nov 02 09:57:19.150253 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.150225    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:19.233280 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.233248    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-workload-certs\") pod \"f58af013-7d2b-4a6b-a55f-e056148cd172\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") "
Nov 02 09:57:19.233471 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.233445    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58af013-7d2b-4a6b-a55f-e056148cd172-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58af013-7d2b-4a6b-a55f-e056148cd172-istiod-ca-cert\") pod \"f58af013-7d2b-4a6b-a55f-e056148cd172\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") "
Nov 02 09:57:19.233539 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.233508    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-workload-socket\") pod \"f58af013-7d2b-4a6b-a55f-e056148cd172\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") "
Nov 02 09:57:19.233580 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.233545    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-data\") pod \"f58af013-7d2b-4a6b-a55f-e056148cd172\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") "
Nov 02 09:57:19.233627 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.233584    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-workload-certs" pod "f58af013-7d2b-4a6b-a55f-e056148cd172" (UID: "f58af013-7d2b-4a6b-a55f-e056148cd172"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:57:19.233662 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.233650    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58af013-7d2b-4a6b-a55f-e056148cd172-kube-api-access-7btxg\" (UniqueName: \"kubernetes.io/projected/f58af013-7d2b-4a6b-a55f-e056148cd172-kube-api-access-7btxg\") pod \"f58af013-7d2b-4a6b-a55f-e056148cd172\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") "
Nov 02 09:57:19.233712 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.233696    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-credential-socket\") pod \"f58af013-7d2b-4a6b-a55f-e056148cd172\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") "
Nov 02 09:57:19.233758 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.233743    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-podinfo\") pod \"f58af013-7d2b-4a6b-a55f-e056148cd172\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") "
Nov 02 09:57:19.233819 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.233777    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-token\" (UniqueName: \"kubernetes.io/projected/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-token\") pod \"f58af013-7d2b-4a6b-a55f-e056148cd172\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") "
Nov 02 09:57:19.233881 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.233836    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-workload-socket" pod "f58af013-7d2b-4a6b-a55f-e056148cd172" (UID: "f58af013-7d2b-4a6b-a55f-e056148cd172"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:57:19.233881 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.233852    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-envoy\") pod \"f58af013-7d2b-4a6b-a55f-e056148cd172\" (UID: \"f58af013-7d2b-4a6b-a55f-e056148cd172\") "
Nov 02 09:57:19.233881 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.233854    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-data" pod "f58af013-7d2b-4a6b-a55f-e056148cd172" (UID: "f58af013-7d2b-4a6b-a55f-e056148cd172"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:57:19.233881 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.233853    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58af013-7d2b-4a6b-a55f-e056148cd172-istiod-ca-cert" pod "f58af013-7d2b-4a6b-a55f-e056148cd172" (UID: "f58af013-7d2b-4a6b-a55f-e056148cd172"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:57:19.234098 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.234033    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-credential-socket" pod "f58af013-7d2b-4a6b-a55f-e056148cd172" (UID: "f58af013-7d2b-4a6b-a55f-e056148cd172"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:57:19.234443 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.234423    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:19.234542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.234446    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58af013-7d2b-4a6b-a55f-e056148cd172-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:19.234542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.234461    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:19.234542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.234473    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:19.234542 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.234486    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:19.235866 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.235845    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-envoy" pod "f58af013-7d2b-4a6b-a55f-e056148cd172" (UID: "f58af013-7d2b-4a6b-a55f-e056148cd172"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:57:19.235940 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.235900    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-podinfo" pod "f58af013-7d2b-4a6b-a55f-e056148cd172" (UID: "f58af013-7d2b-4a6b-a55f-e056148cd172"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:57:19.236240 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.236223    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-token" pod "f58af013-7d2b-4a6b-a55f-e056148cd172" (UID: "f58af013-7d2b-4a6b-a55f-e056148cd172"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:57:19.236309 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.236290    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58af013-7d2b-4a6b-a55f-e056148cd172-kube-api-access-7btxg" pod "f58af013-7d2b-4a6b-a55f-e056148cd172" (UID: "f58af013-7d2b-4a6b-a55f-e056148cd172"). InnerVolumeSpecName "kube-api-access-7btxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:57:19.335227 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.335195    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7btxg\" (UniqueName: \"kubernetes.io/projected/f58af013-7d2b-4a6b-a55f-e056148cd172-kube-api-access-7btxg\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:19.335227 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.335222    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:19.335227 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.335231    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:19.335456 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.335240    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58af013-7d2b-4a6b-a55f-e056148cd172-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:19.983874 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.983837    4614 generic.go:356] "Generic (PLEG): container finished" podID="f58af013-7d2b-4a6b-a55f-e056148cd172" containerID="0db784a64c1860f7c4c3fca066c3736bb98f3d322ef68790c87da7af98a05d98" exitCode=0
Nov 02 09:57:19.984046 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.983908    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"
Nov 02 09:57:19.984046 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.983919    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg" event={"ID":"f58af013-7d2b-4a6b-a55f-e056148cd172","Type":"ContainerDied","Data":"0db784a64c1860f7c4c3fca066c3736bb98f3d322ef68790c87da7af98a05d98"}
Nov 02 09:57:19.984046 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.983999    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg" event={"ID":"f58af013-7d2b-4a6b-a55f-e056148cd172","Type":"ContainerDied","Data":"5113edb945359847e7c93837f4eaf747a8c1e1728ed4282224582371ab86ad00"}
Nov 02 09:57:19.984046 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.984022    4614 scope.go:122] "RemoveContainer" containerID="0db784a64c1860f7c4c3fca066c3736bb98f3d322ef68790c87da7af98a05d98"
Nov 02 09:57:19.994755 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.994585    4614 scope.go:122] "RemoveContainer" containerID="0db784a64c1860f7c4c3fca066c3736bb98f3d322ef68790c87da7af98a05d98"
Nov 02 09:57:19.994922 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:57:19.994904    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db784a64c1860f7c4c3fca066c3736bb98f3d322ef68790c87da7af98a05d98\": container with ID starting with 0db784a64c1860f7c4c3fca066c3736bb98f3d322ef68790c87da7af98a05d98 not found: ID does not exist" containerID="0db784a64c1860f7c4c3fca066c3736bb98f3d322ef68790c87da7af98a05d98"
Nov 02 09:57:19.994970 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:19.994931    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db784a64c1860f7c4c3fca066c3736bb98f3d322ef68790c87da7af98a05d98"} err="failed to get container status \"0db784a64c1860f7c4c3fca066c3736bb98f3d322ef68790c87da7af98a05d98\": rpc error: code = NotFound desc = could not find container \"0db784a64c1860f7c4c3fca066c3736bb98f3d322ef68790c87da7af98a05d98\": container with ID starting with 0db784a64c1860f7c4c3fca066c3736bb98f3d322ef68790c87da7af98a05d98 not found: ID does not exist"
Nov 02 09:57:20.006691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:20.006649    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"]
Nov 02 09:57:20.010173 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:20.010148    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-0c5f4e-openshift-default-84f576f67-6k6gg"]
Nov 02 09:57:21.324288 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:21.324246    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58af013-7d2b-4a6b-a55f-e056148cd172" path="/var/lib/kubelet/pods/f58af013-7d2b-4a6b-a55f-e056148cd172/volumes"
Nov 02 09:57:25.686493 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:25.686441    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:57:25.701375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:25.695428    4614 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:57:25.736517 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:25.736467    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:57:25.856480 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:25.856435    4614 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:57:28.570461 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.570413    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46" podUID="b4c19aac-3aca-42e1-bd87-22c2a3a3509b" containerName="main" containerID="cri-o://434e049f9d264193e21e4c7d7a657db360403bb70b9c1f2c1e1709e9b7889908" gracePeriod=45
Nov 02 09:57:28.848458 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.848433    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:28.850749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.850722    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-home\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-home\") pod \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") "
Nov 02 09:57:28.850881 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.850836    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-model-cache\") pod \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") "
Nov 02 09:57:28.850947 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.850884    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-tls-certs\") pod \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") "
Nov 02 09:57:28.851007 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.850986    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-tmp-dir\") pod \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") "
Nov 02 09:57:28.851007 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.850988    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-home" pod "b4c19aac-3aca-42e1-bd87-22c2a3a3509b" (UID: "b4c19aac-3aca-42e1-bd87-22c2a3a3509b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:57:28.851117 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.851057    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-model-cache" pod "b4c19aac-3aca-42e1-bd87-22c2a3a3509b" (UID: "b4c19aac-3aca-42e1-bd87-22c2a3a3509b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:57:28.851117 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.851069    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-dshm\") pod \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") "
Nov 02 09:57:28.851117 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.851100    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-kube-api-access-rfxdb\" (UniqueName: \"kubernetes.io/projected/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-kube-api-access-rfxdb\") pod \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\" (UID: \"b4c19aac-3aca-42e1-bd87-22c2a3a3509b\") "
Nov 02 09:57:28.851258 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.851208    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-tmp-dir" pod "b4c19aac-3aca-42e1-bd87-22c2a3a3509b" (UID: "b4c19aac-3aca-42e1-bd87-22c2a3a3509b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:57:28.851891 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.851867    4614 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-model-cache\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:28.852030 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.851896    4614 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-tmp-dir\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:28.852030 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.851911    4614 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-home\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:28.853254 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.853227    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-tls-certs" pod "b4c19aac-3aca-42e1-bd87-22c2a3a3509b" (UID: "b4c19aac-3aca-42e1-bd87-22c2a3a3509b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:57:28.853441 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.853408    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-kube-api-access-rfxdb" pod "b4c19aac-3aca-42e1-bd87-22c2a3a3509b" (UID: "b4c19aac-3aca-42e1-bd87-22c2a3a3509b"). InnerVolumeSpecName "kube-api-access-rfxdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:57:28.853441 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.853418    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-dshm" pod "b4c19aac-3aca-42e1-bd87-22c2a3a3509b" (UID: "b4c19aac-3aca-42e1-bd87-22c2a3a3509b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:57:28.953069 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.953015    4614 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-tls-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:28.953069 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.953042    4614 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-dshm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:28.953069 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:28.953053    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfxdb\" (UniqueName: \"kubernetes.io/projected/b4c19aac-3aca-42e1-bd87-22c2a3a3509b-kube-api-access-rfxdb\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:57:29.219257 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:29.219217    4614 generic.go:356] "Generic (PLEG): container finished" podID="b4c19aac-3aca-42e1-bd87-22c2a3a3509b" containerID="434e049f9d264193e21e4c7d7a657db360403bb70b9c1f2c1e1709e9b7889908" exitCode=0
Nov 02 09:57:29.219488 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:29.219290    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"
Nov 02 09:57:29.219488 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:29.219302    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46" event={"ID":"b4c19aac-3aca-42e1-bd87-22c2a3a3509b","Type":"ContainerDied","Data":"434e049f9d264193e21e4c7d7a657db360403bb70b9c1f2c1e1709e9b7889908"}
Nov 02 09:57:29.219644 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:29.219489    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46" event={"ID":"b4c19aac-3aca-42e1-bd87-22c2a3a3509b","Type":"ContainerDied","Data":"445d0abff84be2463b0b1e654819dacf2d2517338a9671e3ea861d66fe30df8b"}
Nov 02 09:57:29.219644 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:29.219522    4614 scope.go:122] "RemoveContainer" containerID="434e049f9d264193e21e4c7d7a657db360403bb70b9c1f2c1e1709e9b7889908"
Nov 02 09:57:29.232337 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:29.232321    4614 scope.go:122] "RemoveContainer" containerID="434e049f9d264193e21e4c7d7a657db360403bb70b9c1f2c1e1709e9b7889908"
Nov 02 09:57:29.232658 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:57:29.232639    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"434e049f9d264193e21e4c7d7a657db360403bb70b9c1f2c1e1709e9b7889908\": container with ID starting with 434e049f9d264193e21e4c7d7a657db360403bb70b9c1f2c1e1709e9b7889908 not found: ID does not exist" containerID="434e049f9d264193e21e4c7d7a657db360403bb70b9c1f2c1e1709e9b7889908"
Nov 02 09:57:29.232728 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:29.232666    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"434e049f9d264193e21e4c7d7a657db360403bb70b9c1f2c1e1709e9b7889908"} err="failed to get container status \"434e049f9d264193e21e4c7d7a657db360403bb70b9c1f2c1e1709e9b7889908\": rpc error: code = NotFound desc = could not find container \"434e049f9d264193e21e4c7d7a657db360403bb70b9c1f2c1e1709e9b7889908\": container with ID starting with 434e049f9d264193e21e4c7d7a657db360403bb70b9c1f2c1e1709e9b7889908 not found: ID does not exist"
Nov 02 09:57:29.247285 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:29.247261    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"]
Nov 02 09:57:29.253623 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:29.253589    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-0c5f4e/noten-model-0c5f4e-kserve-7b495865d9-8rf46"]
Nov 02 09:57:29.316691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:29.316662    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c19aac-3aca-42e1-bd87-22c2a3a3509b" path="/var/lib/kubelet/pods/b4c19aac-3aca-42e1-bd87-22c2a3a3509b/volumes"
Nov 02 09:57:38.154483 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:38.154443    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:57:38.155090 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:38.155055    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:57:38.681514 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:38.681471    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:57:38.682155 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:38.682118    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:57:39.908190 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:57:39.908147    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:57:53.195377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:53.172818    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:57:53.195377 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:53.172913    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:57:53.693481 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:53.693440    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 09:57:53.693831 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:57:53.693797    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:58:02.354047 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:02.354013    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"]
Nov 02 09:58:02.364399 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:02.364369    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"]
Nov 02 09:58:02.379707 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:02.379673    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"]
Nov 02 09:58:09.691295 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:58:09.691255    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:58:15.359374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:15.356577    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:58:17.376023 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.375956    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" containerID="cri-o://b6442cc09e0d7703e2562def86e30c153d5ba0929e7771f55504d11cdffaa379" gracePeriod=45
Nov 02 09:58:17.390579 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.390536    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main" containerID="cri-o://27b26f488ef408cd29d6998efc6b2a5dda1e845c940af829a54fb494f144dc74" gracePeriod=45
Nov 02 09:58:17.403823 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.403756    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main" containerID="cri-o://e6eb4054aea731ca7e7f0ad89b67e55ff146bfb5ee06c56f4ada47cc07ced885" gracePeriod=45
Nov 02 09:58:17.548583 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.548548    4614 generic.go:356] "Generic (PLEG): container finished" podID="19dca143-7178-44e5-9e3e-593f99e267c9" containerID="b6442cc09e0d7703e2562def86e30c153d5ba0929e7771f55504d11cdffaa379" exitCode=0
Nov 02 09:58:17.548788 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.548667    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" event={"ID":"19dca143-7178-44e5-9e3e-593f99e267c9","Type":"ContainerDied","Data":"b6442cc09e0d7703e2562def86e30c153d5ba0929e7771f55504d11cdffaa379"}
Nov 02 09:58:17.562910 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.562874    4614 generic.go:356] "Generic (PLEG): container finished" podID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerID="27b26f488ef408cd29d6998efc6b2a5dda1e845c940af829a54fb494f144dc74" exitCode=0
Nov 02 09:58:17.563254 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.563230    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" event={"ID":"ed9eb91a-cdf9-49bf-8b73-329fe059bc42","Type":"ContainerDied","Data":"27b26f488ef408cd29d6998efc6b2a5dda1e845c940af829a54fb494f144dc74"}
Nov 02 09:58:17.566275 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.565817    4614 generic.go:356] "Generic (PLEG): container finished" podID="86af445e-d01b-4537-b31a-1a78b30099c9" containerID="e6eb4054aea731ca7e7f0ad89b67e55ff146bfb5ee06c56f4ada47cc07ced885" exitCode=0
Nov 02 09:58:17.566629 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.566598    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" event={"ID":"86af445e-d01b-4537-b31a-1a78b30099c9","Type":"ContainerDied","Data":"e6eb4054aea731ca7e7f0ad89b67e55ff146bfb5ee06c56f4ada47cc07ced885"}
Nov 02 09:58:17.779419 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.779387    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:58:17.809004 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.808961    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:58:17.814894 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.814864    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:58:17.884825 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.884794    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-dshm\") pod \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") "
Nov 02 09:58:17.885013 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.884944    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-model-cache\") pod \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") "
Nov 02 09:58:17.885013 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.885003    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-kube-api-access-zntwk\" (UniqueName: \"kubernetes.io/projected/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-kube-api-access-zntwk\") pod \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") "
Nov 02 09:58:17.885140 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.885077    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-tmp-dir\") pod \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") "
Nov 02 09:58:17.885200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.885139    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-home\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-home\") pod \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") "
Nov 02 09:58:17.885259 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.885214    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-tls-certs\") pod \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\" (UID: \"ed9eb91a-cdf9-49bf-8b73-329fe059bc42\") "
Nov 02 09:58:17.885259 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.885240    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-model-cache" pod "ed9eb91a-cdf9-49bf-8b73-329fe059bc42" (UID: "ed9eb91a-cdf9-49bf-8b73-329fe059bc42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.885866 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.885451    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-tmp-dir" pod "ed9eb91a-cdf9-49bf-8b73-329fe059bc42" (UID: "ed9eb91a-cdf9-49bf-8b73-329fe059bc42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.885866 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.885524    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-home" pod "ed9eb91a-cdf9-49bf-8b73-329fe059bc42" (UID: "ed9eb91a-cdf9-49bf-8b73-329fe059bc42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.885866 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.885856    4614 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-model-cache\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.886158 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.885878    4614 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-tmp-dir\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.886158 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.885893    4614 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-home\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.887660 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.887625    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-kube-api-access-zntwk" pod "ed9eb91a-cdf9-49bf-8b73-329fe059bc42" (UID: "ed9eb91a-cdf9-49bf-8b73-329fe059bc42"). InnerVolumeSpecName "kube-api-access-zntwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:17.887812 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.887721    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-dshm" pod "ed9eb91a-cdf9-49bf-8b73-329fe059bc42" (UID: "ed9eb91a-cdf9-49bf-8b73-329fe059bc42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.888546 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.888525    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-tls-certs" pod "ed9eb91a-cdf9-49bf-8b73-329fe059bc42" (UID: "ed9eb91a-cdf9-49bf-8b73-329fe059bc42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:58:17.987341 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987302    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-home\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-home\") pod \"19dca143-7178-44e5-9e3e-593f99e267c9\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") "
Nov 02 09:58:17.987565 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987394    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-home\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-home\") pod \"86af445e-d01b-4537-b31a-1a78b30099c9\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") "
Nov 02 09:58:17.987565 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987482    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86af445e-d01b-4537-b31a-1a78b30099c9-kube-api-access-vn4x8\" (UniqueName: \"kubernetes.io/projected/86af445e-d01b-4537-b31a-1a78b30099c9-kube-api-access-vn4x8\") pod \"86af445e-d01b-4537-b31a-1a78b30099c9\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") "
Nov 02 09:58:17.987565 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987550    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-tmp-dir\") pod \"86af445e-d01b-4537-b31a-1a78b30099c9\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") "
Nov 02 09:58:17.987737 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987580    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-dshm\") pod \"19dca143-7178-44e5-9e3e-593f99e267c9\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") "
Nov 02 09:58:17.987737 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987629    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-home" pod "19dca143-7178-44e5-9e3e-593f99e267c9" (UID: "19dca143-7178-44e5-9e3e-593f99e267c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.987737 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987651    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-home" pod "86af445e-d01b-4537-b31a-1a78b30099c9" (UID: "86af445e-d01b-4537-b31a-1a78b30099c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.987737 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987641    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-model-cache\") pod \"19dca143-7178-44e5-9e3e-593f99e267c9\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") "
Nov 02 09:58:17.987951 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987741    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-tmp-dir\") pod \"19dca143-7178-44e5-9e3e-593f99e267c9\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") "
Nov 02 09:58:17.987951 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987812    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-model-cache\") pod \"86af445e-d01b-4537-b31a-1a78b30099c9\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") "
Nov 02 09:58:17.987951 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987813    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-tmp-dir" pod "86af445e-d01b-4537-b31a-1a78b30099c9" (UID: "86af445e-d01b-4537-b31a-1a78b30099c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.987951 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987821    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-model-cache" pod "19dca143-7178-44e5-9e3e-593f99e267c9" (UID: "19dca143-7178-44e5-9e3e-593f99e267c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.987951 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987850    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19dca143-7178-44e5-9e3e-593f99e267c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/19dca143-7178-44e5-9e3e-593f99e267c9-tls-certs\") pod \"19dca143-7178-44e5-9e3e-593f99e267c9\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") "
Nov 02 09:58:17.987951 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987885    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86af445e-d01b-4537-b31a-1a78b30099c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/86af445e-d01b-4537-b31a-1a78b30099c9-tls-certs\") pod \"86af445e-d01b-4537-b31a-1a78b30099c9\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") "
Nov 02 09:58:17.987951 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987935    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19dca143-7178-44e5-9e3e-593f99e267c9-kube-api-access-vjvsf\" (UniqueName: \"kubernetes.io/projected/19dca143-7178-44e5-9e3e-593f99e267c9-kube-api-access-vjvsf\") pod \"19dca143-7178-44e5-9e3e-593f99e267c9\" (UID: \"19dca143-7178-44e5-9e3e-593f99e267c9\") "
Nov 02 09:58:17.988337 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987975    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-dshm\") pod \"86af445e-d01b-4537-b31a-1a78b30099c9\" (UID: \"86af445e-d01b-4537-b31a-1a78b30099c9\") "
Nov 02 09:58:17.988337 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.987996    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-tmp-dir" pod "19dca143-7178-44e5-9e3e-593f99e267c9" (UID: "19dca143-7178-44e5-9e3e-593f99e267c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.988337 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.988010    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-model-cache" pod "86af445e-d01b-4537-b31a-1a78b30099c9" (UID: "86af445e-d01b-4537-b31a-1a78b30099c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.988808 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.988647    4614 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-home\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.988808 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.988677    4614 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-tls-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.988808 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.988694    4614 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-dshm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.988808 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.988710    4614 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-tmp-dir\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.988808 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.988725    4614 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-model-cache\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.988808 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.988739    4614 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-tmp-dir\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.988808 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.988752    4614 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-model-cache\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.988808 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.988768    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zntwk\" (UniqueName: \"kubernetes.io/projected/ed9eb91a-cdf9-49bf-8b73-329fe059bc42-kube-api-access-zntwk\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.988808 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.988783    4614 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-home\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:17.990024 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.989998    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19dca143-7178-44e5-9e3e-593f99e267c9-tls-certs" pod "19dca143-7178-44e5-9e3e-593f99e267c9" (UID: "19dca143-7178-44e5-9e3e-593f99e267c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:58:17.990148 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.990123    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-dshm" pod "19dca143-7178-44e5-9e3e-593f99e267c9" (UID: "19dca143-7178-44e5-9e3e-593f99e267c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:17.990308 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.990286    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86af445e-d01b-4537-b31a-1a78b30099c9-kube-api-access-vn4x8" pod "86af445e-d01b-4537-b31a-1a78b30099c9" (UID: "86af445e-d01b-4537-b31a-1a78b30099c9"). InnerVolumeSpecName "kube-api-access-vn4x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:17.990468 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.990444    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86af445e-d01b-4537-b31a-1a78b30099c9-tls-certs" pod "86af445e-d01b-4537-b31a-1a78b30099c9" (UID: "86af445e-d01b-4537-b31a-1a78b30099c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 09:58:17.990525 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.990465    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dca143-7178-44e5-9e3e-593f99e267c9-kube-api-access-vjvsf" pod "19dca143-7178-44e5-9e3e-593f99e267c9" (UID: "19dca143-7178-44e5-9e3e-593f99e267c9"). InnerVolumeSpecName "kube-api-access-vjvsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:17.990562 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:17.990537    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-dshm" pod "86af445e-d01b-4537-b31a-1a78b30099c9" (UID: "86af445e-d01b-4537-b31a-1a78b30099c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:18.090034 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.089999    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn4x8\" (UniqueName: \"kubernetes.io/projected/86af445e-d01b-4537-b31a-1a78b30099c9-kube-api-access-vn4x8\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:18.090034 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.090030    4614 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19dca143-7178-44e5-9e3e-593f99e267c9-dshm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:18.090034 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.090041    4614 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19dca143-7178-44e5-9e3e-593f99e267c9-tls-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:18.090269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.090050    4614 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86af445e-d01b-4537-b31a-1a78b30099c9-tls-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:18.090269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.090060    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjvsf\" (UniqueName: \"kubernetes.io/projected/19dca143-7178-44e5-9e3e-593f99e267c9-kube-api-access-vjvsf\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:18.090269 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.090068    4614 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86af445e-d01b-4537-b31a-1a78b30099c9-dshm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:18.573900 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.573870    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"
Nov 02 09:58:18.574348 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.573905    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" event={"ID":"19dca143-7178-44e5-9e3e-593f99e267c9","Type":"ContainerDied","Data":"b649e428b6011ce5002e848d53ecd52f6cc50651381fa33fc4226d2792c18028"}
Nov 02 09:58:18.574348 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.573969    4614 scope.go:122] "RemoveContainer" containerID="b6442cc09e0d7703e2562def86e30c153d5ba0929e7771f55504d11cdffaa379"
Nov 02 09:58:18.575482 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.575412    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"
Nov 02 09:58:18.575482 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.575429    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd" event={"ID":"ed9eb91a-cdf9-49bf-8b73-329fe059bc42","Type":"ContainerDied","Data":"b896bc4aa45dd469dfe441e53eb1d92c39c63397ab0baa14a945785439436a61"}
Nov 02 09:58:18.580802 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.580769    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"
Nov 02 09:58:18.580944 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.580805    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz" event={"ID":"86af445e-d01b-4537-b31a-1a78b30099c9","Type":"ContainerDied","Data":"94384f9748d31e98043b06dac2b97f6952b0f5d488164a6262e84c40a8c58349"}
Nov 02 09:58:18.586041 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.586022    4614 scope.go:122] "RemoveContainer" containerID="27b26f488ef408cd29d6998efc6b2a5dda1e845c940af829a54fb494f144dc74"
Nov 02 09:58:18.597162 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.597147    4614 scope.go:122] "RemoveContainer" containerID="e6eb4054aea731ca7e7f0ad89b67e55ff146bfb5ee06c56f4ada47cc07ced885"
Nov 02 09:58:18.603554 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.603533    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"]
Nov 02 09:58:18.610038 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.610019    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf"]
Nov 02 09:58:18.620691 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.620670    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"]
Nov 02 09:58:18.624566 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.624548    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/premium-w0-2e80fb-kserve-66d5b55789-tbjbd"]
Nov 02 09:58:18.636220 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.636196    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"]
Nov 02 09:58:18.642947 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.642899    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2e80fb/simulator-w0-2e80fb-kserve-555977f8d-dh8rz"]
Nov 02 09:58:18.690428 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:18.690388    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2e80fb/distinct-w0-2e80fb-kserve-5bdf9ccb88-b6tcf" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 09:58:19.316672 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:19.316642    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" path="/var/lib/kubelet/pods/19dca143-7178-44e5-9e3e-593f99e267c9/volumes"
Nov 02 09:58:19.317060 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:19.317045    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" path="/var/lib/kubelet/pods/86af445e-d01b-4537-b31a-1a78b30099c9/volumes"
Nov 02 09:58:19.317459 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:19.317446    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" path="/var/lib/kubelet/pods/ed9eb91a-cdf9-49bf-8b73-329fe059bc42/volumes"
Nov 02 09:58:27.023974 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:27.023933    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"]
Nov 02 09:58:27.029203 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:27.029176    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2604498-t9jqt"]
Nov 02 09:58:27.316838 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:27.316764    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03857226-641c-4716-948b-fe13d3b787c3" path="/var/lib/kubelet/pods/03857226-641c-4716-948b-fe13d3b787c3/volumes"
Nov 02 09:58:36.667397 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:36.661980    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"]
Nov 02 09:58:36.667397 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:36.662245    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" podUID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" containerName="istio-proxy" containerID="cri-o://74df00d7c2e5d4f77f192acfb3f666a5e263dd68b9be638a07a7b1434646ab10" gracePeriod=30
Nov 02 09:58:36.678560 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:36.678529    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"]
Nov 02 09:58:36.678823 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:36.678795    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" podUID="48b039b3-0a79-43d3-bd83-347582902302" containerName="istio-proxy" containerID="cri-o://2ef72950187a7cfb442fb2654d2188de5b3d961ecee9dfef68725fa28b1f9241" gracePeriod=30
Nov 02 09:58:36.689981 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:36.689953    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"]
Nov 02 09:58:36.690216 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:36.690192    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62" podUID="1137058f-bb14-42e6-ac55-d2104536ffcc" containerName="istio-proxy" containerID="cri-o://9df7346abfbbe42e93126985c74796bc917a0b91aa0faacf3823be70b18d742a" gracePeriod=30
Nov 02 09:58:39.704656 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:58:39.704566    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:58:40.044253 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:40.043421    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"]
Nov 02 09:58:40.059859 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:40.059797    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-514e2f48-p5v9j"]
Nov 02 09:58:41.316899 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:41.316864    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="338ef3fd-8640-4e22-a44d-ecb1cf4e8626" path="/var/lib/kubelet/pods/338ef3fd-8640-4e22-a44d-ecb1cf4e8626/volumes"
Nov 02 09:58:42.073415 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.073334    4614 generic.go:356] "Generic (PLEG): container finished" podID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" containerID="74df00d7c2e5d4f77f192acfb3f666a5e263dd68b9be638a07a7b1434646ab10" exitCode=0
Nov 02 09:58:42.073609 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.073459    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" event={"ID":"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7","Type":"ContainerDied","Data":"74df00d7c2e5d4f77f192acfb3f666a5e263dd68b9be638a07a7b1434646ab10"}
Nov 02 09:58:42.074745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.074444    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:58:42.075956 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.075933    4614 generic.go:356] "Generic (PLEG): container finished" podID="48b039b3-0a79-43d3-bd83-347582902302" containerID="2ef72950187a7cfb442fb2654d2188de5b3d961ecee9dfef68725fa28b1f9241" exitCode=0
Nov 02 09:58:42.076238 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.076216    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" event={"ID":"48b039b3-0a79-43d3-bd83-347582902302","Type":"ContainerDied","Data":"2ef72950187a7cfb442fb2654d2188de5b3d961ecee9dfef68725fa28b1f9241"}
Nov 02 09:58:42.076427 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.076408    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv" event={"ID":"48b039b3-0a79-43d3-bd83-347582902302","Type":"ContainerDied","Data":"47a6c0cf1bcddf2dd5f53ffd23a1b2f24d8dd82989a362d23124a842c534311e"}
Nov 02 09:58:42.076573 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.076558    4614 scope.go:122] "RemoveContainer" containerID="2ef72950187a7cfb442fb2654d2188de5b3d961ecee9dfef68725fa28b1f9241"
Nov 02 09:58:42.080062 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.079980    4614 generic.go:356] "Generic (PLEG): container finished" podID="1137058f-bb14-42e6-ac55-d2104536ffcc" containerID="9df7346abfbbe42e93126985c74796bc917a0b91aa0faacf3823be70b18d742a" exitCode=0
Nov 02 09:58:42.080062 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.080023    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62" event={"ID":"1137058f-bb14-42e6-ac55-d2104536ffcc","Type":"ContainerDied","Data":"9df7346abfbbe42e93126985c74796bc917a0b91aa0faacf3823be70b18d742a"}
Nov 02 09:58:42.089977 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.089956    4614 scope.go:122] "RemoveContainer" containerID="2ef72950187a7cfb442fb2654d2188de5b3d961ecee9dfef68725fa28b1f9241"
Nov 02 09:58:42.090326 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:58:42.090302    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef72950187a7cfb442fb2654d2188de5b3d961ecee9dfef68725fa28b1f9241\": container with ID starting with 2ef72950187a7cfb442fb2654d2188de5b3d961ecee9dfef68725fa28b1f9241 not found: ID does not exist" containerID="2ef72950187a7cfb442fb2654d2188de5b3d961ecee9dfef68725fa28b1f9241"
Nov 02 09:58:42.090450 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.090338    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef72950187a7cfb442fb2654d2188de5b3d961ecee9dfef68725fa28b1f9241"} err="failed to get container status \"2ef72950187a7cfb442fb2654d2188de5b3d961ecee9dfef68725fa28b1f9241\": rpc error: code = NotFound desc = could not find container \"2ef72950187a7cfb442fb2654d2188de5b3d961ecee9dfef68725fa28b1f9241\": container with ID starting with 2ef72950187a7cfb442fb2654d2188de5b3d961ecee9dfef68725fa28b1f9241 not found: ID does not exist"
Nov 02 09:58:42.123474 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.123447    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:58:42.153266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.148950    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48b039b3-0a79-43d3-bd83-347582902302-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b039b3-0a79-43d3-bd83-347582902302-istio-podinfo\") pod \"48b039b3-0a79-43d3-bd83-347582902302\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") "
Nov 02 09:58:42.153266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.149023    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-istio-data\") pod \"48b039b3-0a79-43d3-bd83-347582902302\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") "
Nov 02 09:58:42.153266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.149114    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-istio-envoy\") pod \"48b039b3-0a79-43d3-bd83-347582902302\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") "
Nov 02 09:58:42.153266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.149147    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-workload-certs\") pod \"48b039b3-0a79-43d3-bd83-347582902302\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") "
Nov 02 09:58:42.153266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.149207    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b039b3-0a79-43d3-bd83-347582902302-istio-token\" (UniqueName: \"kubernetes.io/projected/48b039b3-0a79-43d3-bd83-347582902302-istio-token\") pod \"48b039b3-0a79-43d3-bd83-347582902302\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") "
Nov 02 09:58:42.153266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.149301    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-workload-socket\") pod \"48b039b3-0a79-43d3-bd83-347582902302\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") "
Nov 02 09:58:42.153266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.149330    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b039b3-0a79-43d3-bd83-347582902302-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b039b3-0a79-43d3-bd83-347582902302-istiod-ca-cert\") pod \"48b039b3-0a79-43d3-bd83-347582902302\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") "
Nov 02 09:58:42.153266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.149394    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-credential-socket\") pod \"48b039b3-0a79-43d3-bd83-347582902302\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") "
Nov 02 09:58:42.153266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.149471    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b039b3-0a79-43d3-bd83-347582902302-kube-api-access-5lvqv\" (UniqueName: \"kubernetes.io/projected/48b039b3-0a79-43d3-bd83-347582902302-kube-api-access-5lvqv\") pod \"48b039b3-0a79-43d3-bd83-347582902302\" (UID: \"48b039b3-0a79-43d3-bd83-347582902302\") "
Nov 02 09:58:42.153266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.150573    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-istio-data" pod "48b039b3-0a79-43d3-bd83-347582902302" (UID: "48b039b3-0a79-43d3-bd83-347582902302"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.153266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.150950    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-workload-certs" pod "48b039b3-0a79-43d3-bd83-347582902302" (UID: "48b039b3-0a79-43d3-bd83-347582902302"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.153266 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.152221    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-workload-socket" pod "48b039b3-0a79-43d3-bd83-347582902302" (UID: "48b039b3-0a79-43d3-bd83-347582902302"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.160417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.155760    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b039b3-0a79-43d3-bd83-347582902302-istio-token" pod "48b039b3-0a79-43d3-bd83-347582902302" (UID: "48b039b3-0a79-43d3-bd83-347582902302"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:42.160417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.155780    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b039b3-0a79-43d3-bd83-347582902302-istiod-ca-cert" pod "48b039b3-0a79-43d3-bd83-347582902302" (UID: "48b039b3-0a79-43d3-bd83-347582902302"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:42.160417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.156222    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.160417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.156247    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b039b3-0a79-43d3-bd83-347582902302-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.160417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.156287    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.160417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.156304    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.160417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.156319    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b039b3-0a79-43d3-bd83-347582902302-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.160417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.156658    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-credential-socket" pod "48b039b3-0a79-43d3-bd83-347582902302" (UID: "48b039b3-0a79-43d3-bd83-347582902302"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.160417 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.158157    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b039b3-0a79-43d3-bd83-347582902302-kube-api-access-5lvqv" pod "48b039b3-0a79-43d3-bd83-347582902302" (UID: "48b039b3-0a79-43d3-bd83-347582902302"). InnerVolumeSpecName "kube-api-access-5lvqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:42.170574 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.170540    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48b039b3-0a79-43d3-bd83-347582902302-istio-podinfo" pod "48b039b3-0a79-43d3-bd83-347582902302" (UID: "48b039b3-0a79-43d3-bd83-347582902302"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:42.182637 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.182544    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"]
Nov 02 09:58:42.182882 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.182850    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy" containerID="cri-o://da6a20935ef9e943dd667f209bf77aa43da36b80490e18c05d30d3c469260fdf" gracePeriod=30
Nov 02 09:58:42.191984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.190963    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"]
Nov 02 09:58:42.191984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.191217    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerName="istio-proxy" containerID="cri-o://dc32472ab974d5194562c5f3038f313a8ddb8d3346c712c2a09d7fc254331db2" gracePeriod=30
Nov 02 09:58:42.192531 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.192502    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-istio-envoy" pod "48b039b3-0a79-43d3-bd83-347582902302" (UID: "48b039b3-0a79-43d3-bd83-347582902302"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.195923 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.195904    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"]
Nov 02 09:58:42.196318 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.196262    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy" containerID="cri-o://f001a6c0f4336d5bfcd40ef5f82c91eb7712da674aa4e8f4ba70dd5822227d68" gracePeriod=30
Nov 02 09:58:42.198196 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.198181    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:58:42.199914 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.199895    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"]
Nov 02 09:58:42.200304 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.200264    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" podUID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" containerName="istio-proxy" containerID="cri-o://efddfbc2ab0f1282b4203098b1532f46efc8e4922ada95d46d61cc4612c9ff59" gracePeriod=30
Nov 02 09:58:42.383435 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.381293    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-data\") pod \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") "
Nov 02 09:58:42.407412 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.381343    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-kube-api-access-dwd4s\" (UniqueName: \"kubernetes.io/projected/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-kube-api-access-dwd4s\") pod \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") "
Nov 02 09:58:42.408282 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.408256    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-data\") pod \"1137058f-bb14-42e6-ac55-d2104536ffcc\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") "
Nov 02 09:58:42.408413 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.408393    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-envoy\") pod \"1137058f-bb14-42e6-ac55-d2104536ffcc\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") "
Nov 02 09:58:42.408479 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.408449    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-podinfo\") pod \"1137058f-bb14-42e6-ac55-d2104536ffcc\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") "
Nov 02 09:58:42.408528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.408484    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-workload-socket\") pod \"1137058f-bb14-42e6-ac55-d2104536ffcc\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") "
Nov 02 09:58:42.408528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.408519    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-credential-socket\") pod \"1137058f-bb14-42e6-ac55-d2104536ffcc\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") "
Nov 02 09:58:42.408622 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.408548    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-token\" (UniqueName: \"kubernetes.io/projected/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-token\") pod \"1137058f-bb14-42e6-ac55-d2104536ffcc\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") "
Nov 02 09:58:42.408674 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.408648    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1137058f-bb14-42e6-ac55-d2104536ffcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1137058f-bb14-42e6-ac55-d2104536ffcc-istiod-ca-cert\") pod \"1137058f-bb14-42e6-ac55-d2104536ffcc\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") "
Nov 02 09:58:42.408722 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.408675    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-token\" (UniqueName: \"kubernetes.io/projected/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-token\") pod \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") "
Nov 02 09:58:42.408722 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.408708    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-credential-socket\") pod \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") "
Nov 02 09:58:42.408837 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.408765    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istiod-ca-cert\") pod \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") "
Nov 02 09:58:42.408837 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.408815    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1137058f-bb14-42e6-ac55-d2104536ffcc-kube-api-access-x4w9m\" (UniqueName: \"kubernetes.io/projected/1137058f-bb14-42e6-ac55-d2104536ffcc-kube-api-access-x4w9m\") pod \"1137058f-bb14-42e6-ac55-d2104536ffcc\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") "
Nov 02 09:58:42.408940 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.408887    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-envoy\") pod \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") "
Nov 02 09:58:42.408940 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.408915    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-workload-socket\") pod \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") "
Nov 02 09:58:42.409033 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.408973    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-workload-certs\") pod \"1137058f-bb14-42e6-ac55-d2104536ffcc\" (UID: \"1137058f-bb14-42e6-ac55-d2104536ffcc\") "
Nov 02 09:58:42.409078 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.409047    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-workload-certs\") pod \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") "
Nov 02 09:58:42.409130 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.409078    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-podinfo\") pod \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\" (UID: \"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7\") "
Nov 02 09:58:42.413697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.409734    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.413697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.413240    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lvqv\" (UniqueName: \"kubernetes.io/projected/48b039b3-0a79-43d3-bd83-347582902302-kube-api-access-5lvqv\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.413697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.413265    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b039b3-0a79-43d3-bd83-347582902302-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.413697 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.413280    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b039b3-0a79-43d3-bd83-347582902302-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.415896 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.415868    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-podinfo" pod "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" (UID: "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:42.416287 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.416262    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-data" pod "1137058f-bb14-42e6-ac55-d2104536ffcc" (UID: "1137058f-bb14-42e6-ac55-d2104536ffcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.418775 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.418743    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-envoy" pod "1137058f-bb14-42e6-ac55-d2104536ffcc" (UID: "1137058f-bb14-42e6-ac55-d2104536ffcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.421961 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.421550    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-podinfo" pod "1137058f-bb14-42e6-ac55-d2104536ffcc" (UID: "1137058f-bb14-42e6-ac55-d2104536ffcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:42.422128 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.422105    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-workload-socket" pod "1137058f-bb14-42e6-ac55-d2104536ffcc" (UID: "1137058f-bb14-42e6-ac55-d2104536ffcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.422303 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.422286    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-credential-socket" pod "1137058f-bb14-42e6-ac55-d2104536ffcc" (UID: "1137058f-bb14-42e6-ac55-d2104536ffcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.424494 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.424467    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-data" pod "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" (UID: "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.426050 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.425256    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-workload-socket" pod "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" (UID: "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.427791 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.426946    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-credential-socket" pod "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" (UID: "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.427791 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.427189    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istiod-ca-cert" pod "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" (UID: "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:42.427791 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.427247    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1137058f-bb14-42e6-ac55-d2104536ffcc-kube-api-access-x4w9m" pod "1137058f-bb14-42e6-ac55-d2104536ffcc" (UID: "1137058f-bb14-42e6-ac55-d2104536ffcc"). InnerVolumeSpecName "kube-api-access-x4w9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:42.427791 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.425233    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-token" pod "1137058f-bb14-42e6-ac55-d2104536ffcc" (UID: "1137058f-bb14-42e6-ac55-d2104536ffcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:42.427791 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.425567    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-workload-certs" pod "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" (UID: "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.427791 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.425844    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1137058f-bb14-42e6-ac55-d2104536ffcc-istiod-ca-cert" pod "1137058f-bb14-42e6-ac55-d2104536ffcc" (UID: "1137058f-bb14-42e6-ac55-d2104536ffcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:42.428665 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.428643    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-workload-certs" pod "1137058f-bb14-42e6-ac55-d2104536ffcc" (UID: "1137058f-bb14-42e6-ac55-d2104536ffcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.445885 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.445831    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-envoy" pod "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" (UID: "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:42.446049 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.445909    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-kube-api-access-dwd4s" pod "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" (UID: "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7"). InnerVolumeSpecName "kube-api-access-dwd4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:42.446948 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.446916    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-token" pod "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" (UID: "6421f1c9-7fdd-4cb1-8313-6c9191cb69f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:42.514804 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514773    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514804 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514801    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514813    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514822    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514834    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514843    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514851    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwd4s\" (UniqueName: \"kubernetes.io/projected/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-kube-api-access-dwd4s\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514861    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514869    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514878    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514886    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514895    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1137058f-bb14-42e6-ac55-d2104536ffcc-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514904    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1137058f-bb14-42e6-ac55-d2104536ffcc-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514912    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1137058f-bb14-42e6-ac55-d2104536ffcc-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514919    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514927    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514935    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:42.514998 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:42.514943    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4w9m\" (UniqueName: \"kubernetes.io/projected/1137058f-bb14-42e6-ac55-d2104536ffcc-kube-api-access-x4w9m\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:43.088012 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.087986    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"
Nov 02 09:58:43.088199 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.088038    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62" event={"ID":"1137058f-bb14-42e6-ac55-d2104536ffcc","Type":"ContainerDied","Data":"b8743c6be3eb9e54123dd15b95a1b9be2aa1c6efa58b5a3d90694ff92ab7bd2d"}
Nov 02 09:58:43.088199 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.088116    4614 scope.go:122] "RemoveContainer" containerID="9df7346abfbbe42e93126985c74796bc917a0b91aa0faacf3823be70b18d742a"
Nov 02 09:58:43.089801 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.089653    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"
Nov 02 09:58:43.089801 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.089714    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt" event={"ID":"6421f1c9-7fdd-4cb1-8313-6c9191cb69f7","Type":"ContainerDied","Data":"86f1eda99e4a796b89a4a52003db419fa4e8a64d6ceda03b1b2e453749e95f6e"}
Nov 02 09:58:43.090954 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.090935    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"
Nov 02 09:58:43.099485 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.099468    4614 scope.go:122] "RemoveContainer" containerID="74df00d7c2e5d4f77f192acfb3f666a5e263dd68b9be638a07a7b1434646ab10"
Nov 02 09:58:43.119403 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.119373    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"]
Nov 02 09:58:43.126519 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.126495    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-6f8mt"]
Nov 02 09:58:43.137801 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.137778    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"]
Nov 02 09:58:43.140723 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.140693    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-4cf62"]
Nov 02 09:58:43.184437 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.184400    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"]
Nov 02 09:58:43.197616 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.197573    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-bde944-openshift-default-75487f99f9-w22hv"]
Nov 02 09:58:43.316478 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.316447    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1137058f-bb14-42e6-ac55-d2104536ffcc" path="/var/lib/kubelet/pods/1137058f-bb14-42e6-ac55-d2104536ffcc/volumes"
Nov 02 09:58:43.316947 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.316934    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b039b3-0a79-43d3-bd83-347582902302" path="/var/lib/kubelet/pods/48b039b3-0a79-43d3-bd83-347582902302/volumes"
Nov 02 09:58:43.317364 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:43.317342    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" path="/var/lib/kubelet/pods/6421f1c9-7fdd-4cb1-8313-6c9191cb69f7/volumes"
Nov 02 09:58:46.296969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:46.296924    4614 patch_prober.go:29] interesting pod/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 09:58:46.297404 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:46.296994    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 09:58:47.542284 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.542250    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:58:47.577582 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.573884    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:58:47.614809 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.614776    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-envoy\") pod \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") "
Nov 02 09:58:47.614945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.614843    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-workload-certs\") pod \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") "
Nov 02 09:58:47.614945 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.614881    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-podinfo\") pod \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") "
Nov 02 09:58:47.615072 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.614952    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-token\" (UniqueName: \"kubernetes.io/projected/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-token\") pod \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") "
Nov 02 09:58:47.615072 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.615007    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-workload-socket\") pod \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") "
Nov 02 09:58:47.615190 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.615077    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-kube-api-access-9h94g\" (UniqueName: \"kubernetes.io/projected/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-kube-api-access-9h94g\") pod \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") "
Nov 02 09:58:47.615190 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.615101    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-workload-certs" pod "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" (UID: "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.615190 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.615112    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istiod-ca-cert\") pod \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") "
Nov 02 09:58:47.615430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.615218    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-data\") pod \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") "
Nov 02 09:58:47.615430 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.615243    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-credential-socket\") pod \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\" (UID: \"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6\") "
Nov 02 09:58:47.615537 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.615490    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istiod-ca-cert" pod "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" (UID: "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:47.615682 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.615666    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-credential-socket" pod "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" (UID: "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.615799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.615781    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.615860 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.615796    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-data" pod "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" (UID: "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.615860 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.615801    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.615860 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.615824    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.616025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.615948    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-workload-socket" pod "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" (UID: "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.618410 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.618384    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-token" pod "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" (UID: "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:47.618650 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.618626    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-podinfo" pod "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" (UID: "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:47.618770 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.618744    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-envoy" pod "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" (UID: "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.620152 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.620132    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-kube-api-access-9h94g" pod "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" (UID: "858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6"). InnerVolumeSpecName "kube-api-access-9h94g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:47.623169 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.623150    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:58:47.716566 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.716520    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istiod-ca-cert\") pod \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") "
Nov 02 09:58:47.716699 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.716602    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/839be18b-58e9-404a-8dfa-3a85e612879d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839be18b-58e9-404a-8dfa-3a85e612879d-istio-podinfo\") pod \"839be18b-58e9-404a-8dfa-3a85e612879d\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") "
Nov 02 09:58:47.716699 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.716679    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-workload-socket\") pod \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") "
Nov 02 09:58:47.716817 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.716730    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-workload-certs\") pod \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") "
Nov 02 09:58:47.716817 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.716760    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-envoy\") pod \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") "
Nov 02 09:58:47.716817 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.716806    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-token\" (UniqueName: \"kubernetes.io/projected/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-token\") pod \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") "
Nov 02 09:58:47.716980 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.716851    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839be18b-58e9-404a-8dfa-3a85e612879d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839be18b-58e9-404a-8dfa-3a85e612879d-istiod-ca-cert\") pod \"839be18b-58e9-404a-8dfa-3a85e612879d\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") "
Nov 02 09:58:47.716980 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.716867    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istiod-ca-cert" pod "d1c147ae-1d41-49a8-89f9-adf158a7ae5c" (UID: "d1c147ae-1d41-49a8-89f9-adf158a7ae5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:47.716980 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.716882    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-workload-socket\") pod \"839be18b-58e9-404a-8dfa-3a85e612879d\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") "
Nov 02 09:58:47.716980 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.716911    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-credential-socket\") pod \"839be18b-58e9-404a-8dfa-3a85e612879d\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") "
Nov 02 09:58:47.717200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717175    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-workload-certs" pod "d1c147ae-1d41-49a8-89f9-adf158a7ae5c" (UID: "d1c147ae-1d41-49a8-89f9-adf158a7ae5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.717200 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717192    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-credential-socket" pod "839be18b-58e9-404a-8dfa-3a85e612879d" (UID: "839be18b-58e9-404a-8dfa-3a85e612879d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.717465 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717395    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-workload-socket" pod "839be18b-58e9-404a-8dfa-3a85e612879d" (UID: "839be18b-58e9-404a-8dfa-3a85e612879d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.717465 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717427    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-workload-socket" pod "d1c147ae-1d41-49a8-89f9-adf158a7ae5c" (UID: "d1c147ae-1d41-49a8-89f9-adf158a7ae5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.717617 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717463    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-kube-api-access-6fv4x\" (UniqueName: \"kubernetes.io/projected/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-kube-api-access-6fv4x\") pod \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") "
Nov 02 09:58:47.717617 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717484    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839be18b-58e9-404a-8dfa-3a85e612879d-istiod-ca-cert" pod "839be18b-58e9-404a-8dfa-3a85e612879d" (UID: "839be18b-58e9-404a-8dfa-3a85e612879d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:47.717617 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717540    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-istio-data\") pod \"839be18b-58e9-404a-8dfa-3a85e612879d\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") "
Nov 02 09:58:47.717617 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717584    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839be18b-58e9-404a-8dfa-3a85e612879d-kube-api-access-5j6f7\" (UniqueName: \"kubernetes.io/projected/839be18b-58e9-404a-8dfa-3a85e612879d-kube-api-access-5j6f7\") pod \"839be18b-58e9-404a-8dfa-3a85e612879d\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") "
Nov 02 09:58:47.717862 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717645    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-podinfo\") pod \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") "
Nov 02 09:58:47.717862 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717687    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-istio-envoy\") pod \"839be18b-58e9-404a-8dfa-3a85e612879d\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") "
Nov 02 09:58:47.717862 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717759    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-workload-certs\") pod \"839be18b-58e9-404a-8dfa-3a85e612879d\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") "
Nov 02 09:58:47.717862 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717797    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-data\") pod \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") "
Nov 02 09:58:47.717862 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717831    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-credential-socket\") pod \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\" (UID: \"d1c147ae-1d41-49a8-89f9-adf158a7ae5c\") "
Nov 02 09:58:47.717862 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717861    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839be18b-58e9-404a-8dfa-3a85e612879d-istio-token\" (UniqueName: \"kubernetes.io/projected/839be18b-58e9-404a-8dfa-3a85e612879d-istio-token\") pod \"839be18b-58e9-404a-8dfa-3a85e612879d\" (UID: \"839be18b-58e9-404a-8dfa-3a85e612879d\") "
Nov 02 09:58:47.718188 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.717866    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-istio-data" pod "839be18b-58e9-404a-8dfa-3a85e612879d" (UID: "839be18b-58e9-404a-8dfa-3a85e612879d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718267    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-workload-certs" pod "839be18b-58e9-404a-8dfa-3a85e612879d" (UID: "839be18b-58e9-404a-8dfa-3a85e612879d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718450    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718504    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718524    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718541    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718557    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718573    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718588    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h94g\" (UniqueName: \"kubernetes.io/projected/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-kube-api-access-9h94g\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718604    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718617    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718628    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718641    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718656    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/839be18b-58e9-404a-8dfa-3a85e612879d-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718547    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-credential-socket" pod "d1c147ae-1d41-49a8-89f9-adf158a7ae5c" (UID: "d1c147ae-1d41-49a8-89f9-adf158a7ae5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.718686 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718671    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.719607 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.718721    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.719607 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.719087    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/839be18b-58e9-404a-8dfa-3a85e612879d-istio-podinfo" pod "839be18b-58e9-404a-8dfa-3a85e612879d" (UID: "839be18b-58e9-404a-8dfa-3a85e612879d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:47.719607 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.719117    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-data" pod "d1c147ae-1d41-49a8-89f9-adf158a7ae5c" (UID: "d1c147ae-1d41-49a8-89f9-adf158a7ae5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.719607 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.719319    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-token" pod "d1c147ae-1d41-49a8-89f9-adf158a7ae5c" (UID: "d1c147ae-1d41-49a8-89f9-adf158a7ae5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:47.719607 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.719435    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-envoy" pod "d1c147ae-1d41-49a8-89f9-adf158a7ae5c" (UID: "d1c147ae-1d41-49a8-89f9-adf158a7ae5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.721035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.721010    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-podinfo" pod "d1c147ae-1d41-49a8-89f9-adf158a7ae5c" (UID: "d1c147ae-1d41-49a8-89f9-adf158a7ae5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:47.721165 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.721143    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839be18b-58e9-404a-8dfa-3a85e612879d-istio-token" pod "839be18b-58e9-404a-8dfa-3a85e612879d" (UID: "839be18b-58e9-404a-8dfa-3a85e612879d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:47.721400 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.721369    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-kube-api-access-6fv4x" pod "d1c147ae-1d41-49a8-89f9-adf158a7ae5c" (UID: "d1c147ae-1d41-49a8-89f9-adf158a7ae5c"). InnerVolumeSpecName "kube-api-access-6fv4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:47.722042 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.722017    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-istio-envoy" pod "839be18b-58e9-404a-8dfa-3a85e612879d" (UID: "839be18b-58e9-404a-8dfa-3a85e612879d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:47.722498 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.722469    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839be18b-58e9-404a-8dfa-3a85e612879d-kube-api-access-5j6f7" pod "839be18b-58e9-404a-8dfa-3a85e612879d" (UID: "839be18b-58e9-404a-8dfa-3a85e612879d"). InnerVolumeSpecName "kube-api-access-5j6f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:47.820324 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.820286    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/839be18b-58e9-404a-8dfa-3a85e612879d-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.820528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.820328    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.820528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.820344    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.820528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.820376    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fv4x\" (UniqueName: \"kubernetes.io/projected/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-kube-api-access-6fv4x\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.820528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.820409    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j6f7\" (UniqueName: \"kubernetes.io/projected/839be18b-58e9-404a-8dfa-3a85e612879d-kube-api-access-5j6f7\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.820528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.820424    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.820528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.820438    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/839be18b-58e9-404a-8dfa-3a85e612879d-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.820528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.820450    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.820528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.820466    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c147ae-1d41-49a8-89f9-adf158a7ae5c-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.820528 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.820481    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/839be18b-58e9-404a-8dfa-3a85e612879d-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:47.881054 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:47.881029    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:58:48.022512 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.022481    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b807678e-0dc1-4d67-9ead-83414e09f53d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b807678e-0dc1-4d67-9ead-83414e09f53d-istiod-ca-cert\") pod \"b807678e-0dc1-4d67-9ead-83414e09f53d\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") "
Nov 02 09:58:48.022685 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.022542    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-workload-certs\") pod \"b807678e-0dc1-4d67-9ead-83414e09f53d\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") "
Nov 02 09:58:48.022685 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.022566    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-workload-socket\") pod \"b807678e-0dc1-4d67-9ead-83414e09f53d\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") "
Nov 02 09:58:48.022685 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.022625    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-podinfo\") pod \"b807678e-0dc1-4d67-9ead-83414e09f53d\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") "
Nov 02 09:58:48.022685 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.022643    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-token\" (UniqueName: \"kubernetes.io/projected/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-token\") pod \"b807678e-0dc1-4d67-9ead-83414e09f53d\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") "
Nov 02 09:58:48.022891 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.022736    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-envoy\") pod \"b807678e-0dc1-4d67-9ead-83414e09f53d\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") "
Nov 02 09:58:48.022891 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.022784    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-data\") pod \"b807678e-0dc1-4d67-9ead-83414e09f53d\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") "
Nov 02 09:58:48.022891 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.022818    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b807678e-0dc1-4d67-9ead-83414e09f53d-kube-api-access-9vpw9\" (UniqueName: \"kubernetes.io/projected/b807678e-0dc1-4d67-9ead-83414e09f53d-kube-api-access-9vpw9\") pod \"b807678e-0dc1-4d67-9ead-83414e09f53d\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") "
Nov 02 09:58:48.022891 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.022829    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-workload-certs" pod "b807678e-0dc1-4d67-9ead-83414e09f53d" (UID: "b807678e-0dc1-4d67-9ead-83414e09f53d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:48.023097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.022906    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-credential-socket\") pod \"b807678e-0dc1-4d67-9ead-83414e09f53d\" (UID: \"b807678e-0dc1-4d67-9ead-83414e09f53d\") "
Nov 02 09:58:48.023097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.022978    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-workload-socket" pod "b807678e-0dc1-4d67-9ead-83414e09f53d" (UID: "b807678e-0dc1-4d67-9ead-83414e09f53d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:48.023097 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.023066    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-data" pod "b807678e-0dc1-4d67-9ead-83414e09f53d" (UID: "b807678e-0dc1-4d67-9ead-83414e09f53d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:48.023312 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.023287    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-credential-socket" pod "b807678e-0dc1-4d67-9ead-83414e09f53d" (UID: "b807678e-0dc1-4d67-9ead-83414e09f53d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:48.023387 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.023308    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b807678e-0dc1-4d67-9ead-83414e09f53d-istiod-ca-cert" pod "b807678e-0dc1-4d67-9ead-83414e09f53d" (UID: "b807678e-0dc1-4d67-9ead-83414e09f53d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:48.023484 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.023464    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:48.023611 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.023486    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:48.023611 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.023501    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b807678e-0dc1-4d67-9ead-83414e09f53d-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:48.023611 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.023514    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:48.023611 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.023528    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:48.024956 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.024929    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-podinfo" pod "b807678e-0dc1-4d67-9ead-83414e09f53d" (UID: "b807678e-0dc1-4d67-9ead-83414e09f53d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:48.025064 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.025037    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-token" pod "b807678e-0dc1-4d67-9ead-83414e09f53d" (UID: "b807678e-0dc1-4d67-9ead-83414e09f53d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:48.025148 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.025129    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b807678e-0dc1-4d67-9ead-83414e09f53d-kube-api-access-9vpw9" pod "b807678e-0dc1-4d67-9ead-83414e09f53d" (UID: "b807678e-0dc1-4d67-9ead-83414e09f53d"). InnerVolumeSpecName "kube-api-access-9vpw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:48.025210 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.025196    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-envoy" pod "b807678e-0dc1-4d67-9ead-83414e09f53d" (UID: "b807678e-0dc1-4d67-9ead-83414e09f53d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:48.124765 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.124687    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:48.124765 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.124720    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vpw9\" (UniqueName: \"kubernetes.io/projected/b807678e-0dc1-4d67-9ead-83414e09f53d-kube-api-access-9vpw9\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:48.124765 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.124730    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:48.124765 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.124738    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b807678e-0dc1-4d67-9ead-83414e09f53d-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:48.128730 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.128702    4614 generic.go:356] "Generic (PLEG): container finished" podID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerID="dc32472ab974d5194562c5f3038f313a8ddb8d3346c712c2a09d7fc254331db2" exitCode=0
Nov 02 09:58:48.128877 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.128751    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"
Nov 02 09:58:48.128877 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.128796    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" event={"ID":"d1c147ae-1d41-49a8-89f9-adf158a7ae5c","Type":"ContainerDied","Data":"dc32472ab974d5194562c5f3038f313a8ddb8d3346c712c2a09d7fc254331db2"}
Nov 02 09:58:48.128877 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.128857    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw" event={"ID":"d1c147ae-1d41-49a8-89f9-adf158a7ae5c","Type":"ContainerDied","Data":"b0d600ee8380d861e2ba2c69c66f16d03dd430f3c584931eaae80ef6d2b2cc67"}
Nov 02 09:58:48.129028 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.128885    4614 scope.go:122] "RemoveContainer" containerID="dc32472ab974d5194562c5f3038f313a8ddb8d3346c712c2a09d7fc254331db2"
Nov 02 09:58:48.130303 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.130275    4614 generic.go:356] "Generic (PLEG): container finished" podID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerID="f001a6c0f4336d5bfcd40ef5f82c91eb7712da674aa4e8f4ba70dd5822227d68" exitCode=0
Nov 02 09:58:48.130470 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.130315    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"
Nov 02 09:58:48.130470 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.130385    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" event={"ID":"b807678e-0dc1-4d67-9ead-83414e09f53d","Type":"ContainerDied","Data":"f001a6c0f4336d5bfcd40ef5f82c91eb7712da674aa4e8f4ba70dd5822227d68"}
Nov 02 09:58:48.130470 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.130437    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7" event={"ID":"b807678e-0dc1-4d67-9ead-83414e09f53d","Type":"ContainerDied","Data":"16c4cd2403881c32669b49f906c6b05b43f432e884c8e7e0474bdeed02f89b24"}
Nov 02 09:58:48.131799 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.131780    4614 generic.go:356] "Generic (PLEG): container finished" podID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" containerID="efddfbc2ab0f1282b4203098b1532f46efc8e4922ada95d46d61cc4612c9ff59" exitCode=0
Nov 02 09:58:48.131909 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.131827    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"
Nov 02 09:58:48.131909 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.131862    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" event={"ID":"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6","Type":"ContainerDied","Data":"efddfbc2ab0f1282b4203098b1532f46efc8e4922ada95d46d61cc4612c9ff59"}
Nov 02 09:58:48.132040 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.131909    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x" event={"ID":"858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6","Type":"ContainerDied","Data":"1ead096d1b31bd93e966ab9c7f7bdc42492744b76e00aadfaf820cd72831bebc"}
Nov 02 09:58:48.133984 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.133962    4614 generic.go:356] "Generic (PLEG): container finished" podID="839be18b-58e9-404a-8dfa-3a85e612879d" containerID="da6a20935ef9e943dd667f209bf77aa43da36b80490e18c05d30d3c469260fdf" exitCode=0
Nov 02 09:58:48.134054 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.134001    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" event={"ID":"839be18b-58e9-404a-8dfa-3a85e612879d","Type":"ContainerDied","Data":"da6a20935ef9e943dd667f209bf77aa43da36b80490e18c05d30d3c469260fdf"}
Nov 02 09:58:48.134054 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.134039    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl" event={"ID":"839be18b-58e9-404a-8dfa-3a85e612879d","Type":"ContainerDied","Data":"bde8237f641e96e3074febd6c0d6c0f0a98c600da379d0c7d38196a9074c1047"}
Nov 02 09:58:48.134144 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.134003    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"
Nov 02 09:58:48.143922 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.143901    4614 scope.go:122] "RemoveContainer" containerID="dc32472ab974d5194562c5f3038f313a8ddb8d3346c712c2a09d7fc254331db2"
Nov 02 09:58:48.144198 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:58:48.144179    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc32472ab974d5194562c5f3038f313a8ddb8d3346c712c2a09d7fc254331db2\": container with ID starting with dc32472ab974d5194562c5f3038f313a8ddb8d3346c712c2a09d7fc254331db2 not found: ID does not exist" containerID="dc32472ab974d5194562c5f3038f313a8ddb8d3346c712c2a09d7fc254331db2"
Nov 02 09:58:48.144251 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.144205    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc32472ab974d5194562c5f3038f313a8ddb8d3346c712c2a09d7fc254331db2"} err="failed to get container status \"dc32472ab974d5194562c5f3038f313a8ddb8d3346c712c2a09d7fc254331db2\": rpc error: code = NotFound desc = could not find container \"dc32472ab974d5194562c5f3038f313a8ddb8d3346c712c2a09d7fc254331db2\": container with ID starting with dc32472ab974d5194562c5f3038f313a8ddb8d3346c712c2a09d7fc254331db2 not found: ID does not exist"
Nov 02 09:58:48.144251 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.144221    4614 scope.go:122] "RemoveContainer" containerID="f001a6c0f4336d5bfcd40ef5f82c91eb7712da674aa4e8f4ba70dd5822227d68"
Nov 02 09:58:48.156022 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.156005    4614 scope.go:122] "RemoveContainer" containerID="f001a6c0f4336d5bfcd40ef5f82c91eb7712da674aa4e8f4ba70dd5822227d68"
Nov 02 09:58:48.156284 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:58:48.156267    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f001a6c0f4336d5bfcd40ef5f82c91eb7712da674aa4e8f4ba70dd5822227d68\": container with ID starting with f001a6c0f4336d5bfcd40ef5f82c91eb7712da674aa4e8f4ba70dd5822227d68 not found: ID does not exist" containerID="f001a6c0f4336d5bfcd40ef5f82c91eb7712da674aa4e8f4ba70dd5822227d68"
Nov 02 09:58:48.156331 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.156290    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f001a6c0f4336d5bfcd40ef5f82c91eb7712da674aa4e8f4ba70dd5822227d68"} err="failed to get container status \"f001a6c0f4336d5bfcd40ef5f82c91eb7712da674aa4e8f4ba70dd5822227d68\": rpc error: code = NotFound desc = could not find container \"f001a6c0f4336d5bfcd40ef5f82c91eb7712da674aa4e8f4ba70dd5822227d68\": container with ID starting with f001a6c0f4336d5bfcd40ef5f82c91eb7712da674aa4e8f4ba70dd5822227d68 not found: ID does not exist"
Nov 02 09:58:48.156331 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.156305    4614 scope.go:122] "RemoveContainer" containerID="efddfbc2ab0f1282b4203098b1532f46efc8e4922ada95d46d61cc4612c9ff59"
Nov 02 09:58:48.165337 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.165308    4614 scope.go:122] "RemoveContainer" containerID="efddfbc2ab0f1282b4203098b1532f46efc8e4922ada95d46d61cc4612c9ff59"
Nov 02 09:58:48.165618 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:58:48.165588    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efddfbc2ab0f1282b4203098b1532f46efc8e4922ada95d46d61cc4612c9ff59\": container with ID starting with efddfbc2ab0f1282b4203098b1532f46efc8e4922ada95d46d61cc4612c9ff59 not found: ID does not exist" containerID="efddfbc2ab0f1282b4203098b1532f46efc8e4922ada95d46d61cc4612c9ff59"
Nov 02 09:58:48.165675 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.165626    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efddfbc2ab0f1282b4203098b1532f46efc8e4922ada95d46d61cc4612c9ff59"} err="failed to get container status \"efddfbc2ab0f1282b4203098b1532f46efc8e4922ada95d46d61cc4612c9ff59\": rpc error: code = NotFound desc = could not find container \"efddfbc2ab0f1282b4203098b1532f46efc8e4922ada95d46d61cc4612c9ff59\": container with ID starting with efddfbc2ab0f1282b4203098b1532f46efc8e4922ada95d46d61cc4612c9ff59 not found: ID does not exist"
Nov 02 09:58:48.165675 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.165645    4614 scope.go:122] "RemoveContainer" containerID="da6a20935ef9e943dd667f209bf77aa43da36b80490e18c05d30d3c469260fdf"
Nov 02 09:58:48.166899 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.166285    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"]
Nov 02 09:58:48.226375 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.200000    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-bcqxw"]
Nov 02 09:58:48.250374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.218815    4614 scope.go:122] "RemoveContainer" containerID="da6a20935ef9e943dd667f209bf77aa43da36b80490e18c05d30d3c469260fdf"
Nov 02 09:58:48.250374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.229157    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"]
Nov 02 09:58:48.250374 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:58:48.229600    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6a20935ef9e943dd667f209bf77aa43da36b80490e18c05d30d3c469260fdf\": container with ID starting with da6a20935ef9e943dd667f209bf77aa43da36b80490e18c05d30d3c469260fdf not found: ID does not exist" containerID="da6a20935ef9e943dd667f209bf77aa43da36b80490e18c05d30d3c469260fdf"
Nov 02 09:58:48.250374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.229635    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6a20935ef9e943dd667f209bf77aa43da36b80490e18c05d30d3c469260fdf"} err="failed to get container status \"da6a20935ef9e943dd667f209bf77aa43da36b80490e18c05d30d3c469260fdf\": rpc error: code = NotFound desc = could not find container \"da6a20935ef9e943dd667f209bf77aa43da36b80490e18c05d30d3c469260fdf\": container with ID starting with da6a20935ef9e943dd667f209bf77aa43da36b80490e18c05d30d3c469260fdf not found: ID does not exist"
Nov 02 09:58:48.259382 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.252810    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rmc6x"]
Nov 02 09:58:48.315374 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.313427    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"]
Nov 02 09:58:48.322100 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.316729    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-l2jsl"]
Nov 02 09:58:48.322100 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.320826    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"]
Nov 02 09:58:48.322999 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:48.322973    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2e80fb-openshift-default-5585568575-rg5f7"]
Nov 02 09:58:49.315927 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:49.315894    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" path="/var/lib/kubelet/pods/839be18b-58e9-404a-8dfa-3a85e612879d/volumes"
Nov 02 09:58:49.316344 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:49.316330    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" path="/var/lib/kubelet/pods/858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6/volumes"
Nov 02 09:58:49.316826 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:49.316809    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" path="/var/lib/kubelet/pods/b807678e-0dc1-4d67-9ead-83414e09f53d/volumes"
Nov 02 09:58:49.317566 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:49.317551    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" path="/var/lib/kubelet/pods/d1c147ae-1d41-49a8-89f9-adf158a7ae5c/volumes"
Nov 02 09:58:52.632466 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:52.632429    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"]
Nov 02 09:58:52.632957 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:52.632673    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv" podUID="c94175a6-949c-4995-b110-b9749c2920b6" containerName="istio-proxy" containerID="cri-o://348d499c0112424af3f34d6800c4b29c68b4c09177e08de96541bf3d97ef60e1" gracePeriod=30
Nov 02 09:58:52.639840 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:52.639811    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"]
Nov 02 09:58:52.640238 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:52.640067    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw" podUID="e67b5ee6-666e-4be6-ad46-b0020f405e7e" containerName="istio-proxy" containerID="cri-o://f1fd2f816b8746fead0d363ba09930602160f1df9c607b3c82aa3b4a5e540545" gracePeriod=30
Nov 02 09:58:52.653745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:52.646120    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"]
Nov 02 09:58:52.653745 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:52.646407    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q" podUID="7a55948d-7d73-4560-85fa-88cb221e3e9d" containerName="istio-proxy" containerID="cri-o://c84d182b1e2f5544e44f0607d6fe3eaa3a6bc4ed3b835b78c33460d15ba3f844" gracePeriod=30
Nov 02 09:58:56.027651 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:56.026399    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"]
Nov 02 09:58:56.030388 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:56.029554    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3e1453-7v89m"]
Nov 02 09:58:57.366493 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:57.363752    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b8d319-6b2e-42c2-b7c5-0ead72698cc2" path="/var/lib/kubelet/pods/53b8d319-6b2e-42c2-b7c5-0ead72698cc2/volumes"
Nov 02 09:58:57.945601 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:57.945562    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:58:57.966844 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:57.966819    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:58:58.076491 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.076463    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:58:58.111583 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.111549    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-data\") pod \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") "
Nov 02 09:58:58.111749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.111587    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-envoy\") pod \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") "
Nov 02 09:58:58.111749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.111619    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-istio-envoy\") pod \"c94175a6-949c-4995-b110-b9749c2920b6\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") "
Nov 02 09:58:58.111749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.111672    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-podinfo\") pod \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") "
Nov 02 09:58:58.111749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.111719    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-istio-data\") pod \"c94175a6-949c-4995-b110-b9749c2920b6\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") "
Nov 02 09:58:58.111749 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.111743    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e67b5ee6-666e-4be6-ad46-b0020f405e7e-kube-api-access-nz7d6\" (UniqueName: \"kubernetes.io/projected/e67b5ee6-666e-4be6-ad46-b0020f405e7e-kube-api-access-nz7d6\") pod \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") "
Nov 02 09:58:58.112035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.111778    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-credential-socket\") pod \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") "
Nov 02 09:58:58.112035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.111803    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-credential-socket\") pod \"c94175a6-949c-4995-b110-b9749c2920b6\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") "
Nov 02 09:58:58.112035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.111821    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-workload-certs\") pod \"c94175a6-949c-4995-b110-b9749c2920b6\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") "
Nov 02 09:58:58.112035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.111862    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c94175a6-949c-4995-b110-b9749c2920b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94175a6-949c-4995-b110-b9749c2920b6-istio-podinfo\") pod \"c94175a6-949c-4995-b110-b9749c2920b6\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") "
Nov 02 09:58:58.112035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.111892    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-workload-certs\") pod \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") "
Nov 02 09:58:58.112035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.111911    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istiod-ca-cert\") pod \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") "
Nov 02 09:58:58.112035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.111933    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-workload-socket\") pod \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") "
Nov 02 09:58:58.112035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.111964    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94175a6-949c-4995-b110-b9749c2920b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94175a6-949c-4995-b110-b9749c2920b6-istiod-ca-cert\") pod \"c94175a6-949c-4995-b110-b9749c2920b6\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") "
Nov 02 09:58:58.112035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.112010    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94175a6-949c-4995-b110-b9749c2920b6-kube-api-access-t2n82\" (UniqueName: \"kubernetes.io/projected/c94175a6-949c-4995-b110-b9749c2920b6-kube-api-access-t2n82\") pod \"c94175a6-949c-4995-b110-b9749c2920b6\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") "
Nov 02 09:58:58.112035 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.112030    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-workload-socket\") pod \"c94175a6-949c-4995-b110-b9749c2920b6\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") "
Nov 02 09:58:58.112548 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.112060    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94175a6-949c-4995-b110-b9749c2920b6-istio-token\" (UniqueName: \"kubernetes.io/projected/c94175a6-949c-4995-b110-b9749c2920b6-istio-token\") pod \"c94175a6-949c-4995-b110-b9749c2920b6\" (UID: \"c94175a6-949c-4995-b110-b9749c2920b6\") "
Nov 02 09:58:58.112548 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.112075    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-token\" (UniqueName: \"kubernetes.io/projected/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-token\") pod \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\" (UID: \"e67b5ee6-666e-4be6-ad46-b0020f405e7e\") "
Nov 02 09:58:58.112663 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.112630    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-workload-certs" pod "c94175a6-949c-4995-b110-b9749c2920b6" (UID: "c94175a6-949c-4995-b110-b9749c2920b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.112831 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.112803    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-workload-socket" pod "e67b5ee6-666e-4be6-ad46-b0020f405e7e" (UID: "e67b5ee6-666e-4be6-ad46-b0020f405e7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.112905 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.112815    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-workload-certs" pod "e67b5ee6-666e-4be6-ad46-b0020f405e7e" (UID: "e67b5ee6-666e-4be6-ad46-b0020f405e7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.112969 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.112947    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-istio-data" pod "c94175a6-949c-4995-b110-b9749c2920b6" (UID: "c94175a6-949c-4995-b110-b9749c2920b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.113830 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.113807    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-data" pod "e67b5ee6-666e-4be6-ad46-b0020f405e7e" (UID: "e67b5ee6-666e-4be6-ad46-b0020f405e7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.114137 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.114118    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-credential-socket" pod "c94175a6-949c-4995-b110-b9749c2920b6" (UID: "c94175a6-949c-4995-b110-b9749c2920b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.114444 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.114424    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-workload-socket" pod "c94175a6-949c-4995-b110-b9749c2920b6" (UID: "c94175a6-949c-4995-b110-b9749c2920b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.115055 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.115030    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-credential-socket" pod "e67b5ee6-666e-4be6-ad46-b0020f405e7e" (UID: "e67b5ee6-666e-4be6-ad46-b0020f405e7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.115923 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.115577    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-envoy" pod "e67b5ee6-666e-4be6-ad46-b0020f405e7e" (UID: "e67b5ee6-666e-4be6-ad46-b0020f405e7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.115923 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.115586    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c94175a6-949c-4995-b110-b9749c2920b6-istio-podinfo" pod "c94175a6-949c-4995-b110-b9749c2920b6" (UID: "c94175a6-949c-4995-b110-b9749c2920b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:58.115923 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.115623    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-token" pod "e67b5ee6-666e-4be6-ad46-b0020f405e7e" (UID: "e67b5ee6-666e-4be6-ad46-b0020f405e7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:58.115923 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.115877    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94175a6-949c-4995-b110-b9749c2920b6-kube-api-access-t2n82" pod "c94175a6-949c-4995-b110-b9749c2920b6" (UID: "c94175a6-949c-4995-b110-b9749c2920b6"). InnerVolumeSpecName "kube-api-access-t2n82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:58.116183 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.115924    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94175a6-949c-4995-b110-b9749c2920b6-istiod-ca-cert" pod "c94175a6-949c-4995-b110-b9749c2920b6" (UID: "c94175a6-949c-4995-b110-b9749c2920b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:58.116948 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.116922    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istiod-ca-cert" pod "e67b5ee6-666e-4be6-ad46-b0020f405e7e" (UID: "e67b5ee6-666e-4be6-ad46-b0020f405e7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:58.118028 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.117995    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-podinfo" pod "e67b5ee6-666e-4be6-ad46-b0020f405e7e" (UID: "e67b5ee6-666e-4be6-ad46-b0020f405e7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:58.119526 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.119501    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-istio-envoy" pod "c94175a6-949c-4995-b110-b9749c2920b6" (UID: "c94175a6-949c-4995-b110-b9749c2920b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.126802 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.126769    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94175a6-949c-4995-b110-b9749c2920b6-istio-token" pod "c94175a6-949c-4995-b110-b9749c2920b6" (UID: "c94175a6-949c-4995-b110-b9749c2920b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:58.128038 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.128017    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67b5ee6-666e-4be6-ad46-b0020f405e7e-kube-api-access-nz7d6" pod "e67b5ee6-666e-4be6-ad46-b0020f405e7e" (UID: "e67b5ee6-666e-4be6-ad46-b0020f405e7e"). InnerVolumeSpecName "kube-api-access-nz7d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:58.213060 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.212980    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-workload-certs\") pod \"7a55948d-7d73-4560-85fa-88cb221e3e9d\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") "
Nov 02 09:58:58.213060 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213034    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a55948d-7d73-4560-85fa-88cb221e3e9d-kube-api-access-5ktc9\" (UniqueName: \"kubernetes.io/projected/7a55948d-7d73-4560-85fa-88cb221e3e9d-kube-api-access-5ktc9\") pod \"7a55948d-7d73-4560-85fa-88cb221e3e9d\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") "
Nov 02 09:58:58.213414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213075    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-envoy\") pod \"7a55948d-7d73-4560-85fa-88cb221e3e9d\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") "
Nov 02 09:58:58.213414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213100    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-data\") pod \"7a55948d-7d73-4560-85fa-88cb221e3e9d\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") "
Nov 02 09:58:58.213414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213180    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-credential-socket\") pod \"7a55948d-7d73-4560-85fa-88cb221e3e9d\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") "
Nov 02 09:58:58.213414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213234    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a55948d-7d73-4560-85fa-88cb221e3e9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a55948d-7d73-4560-85fa-88cb221e3e9d-istiod-ca-cert\") pod \"7a55948d-7d73-4560-85fa-88cb221e3e9d\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") "
Nov 02 09:58:58.213414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213325    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-token\" (UniqueName: \"kubernetes.io/projected/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-token\") pod \"7a55948d-7d73-4560-85fa-88cb221e3e9d\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") "
Nov 02 09:58:58.213414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213373    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-workload-socket\") pod \"7a55948d-7d73-4560-85fa-88cb221e3e9d\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") "
Nov 02 09:58:58.213736 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213469    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-podinfo\") pod \"7a55948d-7d73-4560-85fa-88cb221e3e9d\" (UID: \"7a55948d-7d73-4560-85fa-88cb221e3e9d\") "
Nov 02 09:58:58.213736 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213229    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-workload-certs" pod "7a55948d-7d73-4560-85fa-88cb221e3e9d" (UID: "7a55948d-7d73-4560-85fa-88cb221e3e9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.213736 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213463    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-data" pod "7a55948d-7d73-4560-85fa-88cb221e3e9d" (UID: "7a55948d-7d73-4560-85fa-88cb221e3e9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.213736 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213477    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-credential-socket" pod "7a55948d-7d73-4560-85fa-88cb221e3e9d" (UID: "7a55948d-7d73-4560-85fa-88cb221e3e9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.213736 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213674    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-workload-socket" pod "7a55948d-7d73-4560-85fa-88cb221e3e9d" (UID: "7a55948d-7d73-4560-85fa-88cb221e3e9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.213736 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213693    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a55948d-7d73-4560-85fa-88cb221e3e9d-istiod-ca-cert" pod "7a55948d-7d73-4560-85fa-88cb221e3e9d" (UID: "7a55948d-7d73-4560-85fa-88cb221e3e9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 09:58:58.214065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213940    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213964    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213979    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.213994    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94175a6-949c-4995-b110-b9749c2920b6-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214008    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a55948d-7d73-4560-85fa-88cb221e3e9d-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214022    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214035    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214050    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214065 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214064    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94175a6-949c-4995-b110-b9749c2920b6-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214078    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214092    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2n82\" (UniqueName: \"kubernetes.io/projected/c94175a6-949c-4995-b110-b9749c2920b6-kube-api-access-t2n82\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214106    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214120    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94175a6-949c-4995-b110-b9749c2920b6-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214132    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214146    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214159    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214174    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214187    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e67b5ee6-666e-4be6-ad46-b0020f405e7e-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214201    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214215    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94175a6-949c-4995-b110-b9749c2920b6-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214228    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214245    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz7d6\" (UniqueName: \"kubernetes.io/projected/e67b5ee6-666e-4be6-ad46-b0020f405e7e-kube-api-access-nz7d6\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.214414 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.214260    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e67b5ee6-666e-4be6-ad46-b0020f405e7e-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.215206 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.215185    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-envoy" pod "7a55948d-7d73-4560-85fa-88cb221e3e9d" (UID: "7a55948d-7d73-4560-85fa-88cb221e3e9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 09:58:58.215436 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.215418    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-token" pod "7a55948d-7d73-4560-85fa-88cb221e3e9d" (UID: "7a55948d-7d73-4560-85fa-88cb221e3e9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:58.215682 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.215663    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a55948d-7d73-4560-85fa-88cb221e3e9d-kube-api-access-5ktc9" pod "7a55948d-7d73-4560-85fa-88cb221e3e9d" (UID: "7a55948d-7d73-4560-85fa-88cb221e3e9d"). InnerVolumeSpecName "kube-api-access-5ktc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 09:58:58.215776 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.215760    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-podinfo" pod "7a55948d-7d73-4560-85fa-88cb221e3e9d" (UID: "7a55948d-7d73-4560-85fa-88cb221e3e9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 09:58:58.232224 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.232198    4614 generic.go:356] "Generic (PLEG): container finished" podID="e67b5ee6-666e-4be6-ad46-b0020f405e7e" containerID="f1fd2f816b8746fead0d363ba09930602160f1df9c607b3c82aa3b4a5e540545" exitCode=0
Nov 02 09:58:58.232334 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.232254    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"
Nov 02 09:58:58.232334 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.232280    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw" event={"ID":"e67b5ee6-666e-4be6-ad46-b0020f405e7e","Type":"ContainerDied","Data":"f1fd2f816b8746fead0d363ba09930602160f1df9c607b3c82aa3b4a5e540545"}
Nov 02 09:58:58.232475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.232402    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw" event={"ID":"e67b5ee6-666e-4be6-ad46-b0020f405e7e","Type":"ContainerDied","Data":"53a5cb462c56e07af146e61375c0e7be97e73d9ff46e5eb88e8b9416378eb635"}
Nov 02 09:58:58.232475 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.232438    4614 scope.go:122] "RemoveContainer" containerID="f1fd2f816b8746fead0d363ba09930602160f1df9c607b3c82aa3b4a5e540545"
Nov 02 09:58:58.233764 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.233745    4614 generic.go:356] "Generic (PLEG): container finished" podID="c94175a6-949c-4995-b110-b9749c2920b6" containerID="348d499c0112424af3f34d6800c4b29c68b4c09177e08de96541bf3d97ef60e1" exitCode=0
Nov 02 09:58:58.233861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.233798    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"
Nov 02 09:58:58.233861 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.233828    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv" event={"ID":"c94175a6-949c-4995-b110-b9749c2920b6","Type":"ContainerDied","Data":"348d499c0112424af3f34d6800c4b29c68b4c09177e08de96541bf3d97ef60e1"}
Nov 02 09:58:58.233971 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.233873    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv" event={"ID":"c94175a6-949c-4995-b110-b9749c2920b6","Type":"ContainerDied","Data":"7bc3da263472cb4679154333a5d94eaa22b204b19a65cb290d48173d7ade6ec1"}
Nov 02 09:58:58.235418 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.235392    4614 generic.go:356] "Generic (PLEG): container finished" podID="7a55948d-7d73-4560-85fa-88cb221e3e9d" containerID="c84d182b1e2f5544e44f0607d6fe3eaa3a6bc4ed3b835b78c33460d15ba3f844" exitCode=0
Nov 02 09:58:58.235529 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.235432    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q" event={"ID":"7a55948d-7d73-4560-85fa-88cb221e3e9d","Type":"ContainerDied","Data":"c84d182b1e2f5544e44f0607d6fe3eaa3a6bc4ed3b835b78c33460d15ba3f844"}
Nov 02 09:58:58.235529 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.235451    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"
Nov 02 09:58:58.235529 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.235468    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q" event={"ID":"7a55948d-7d73-4560-85fa-88cb221e3e9d","Type":"ContainerDied","Data":"7853ac9dcf282b208100b783ce18215e70e870bcb8af9e8378a8d834fed5ed94"}
Nov 02 09:58:58.243569 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.243552    4614 scope.go:122] "RemoveContainer" containerID="f1fd2f816b8746fead0d363ba09930602160f1df9c607b3c82aa3b4a5e540545"
Nov 02 09:58:58.243850 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:58:58.243834    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1fd2f816b8746fead0d363ba09930602160f1df9c607b3c82aa3b4a5e540545\": container with ID starting with f1fd2f816b8746fead0d363ba09930602160f1df9c607b3c82aa3b4a5e540545 not found: ID does not exist" containerID="f1fd2f816b8746fead0d363ba09930602160f1df9c607b3c82aa3b4a5e540545"
Nov 02 09:58:58.243920 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.243857    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fd2f816b8746fead0d363ba09930602160f1df9c607b3c82aa3b4a5e540545"} err="failed to get container status \"f1fd2f816b8746fead0d363ba09930602160f1df9c607b3c82aa3b4a5e540545\": rpc error: code = NotFound desc = could not find container \"f1fd2f816b8746fead0d363ba09930602160f1df9c607b3c82aa3b4a5e540545\": container with ID starting with f1fd2f816b8746fead0d363ba09930602160f1df9c607b3c82aa3b4a5e540545 not found: ID does not exist"
Nov 02 09:58:58.243920 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.243873    4614 scope.go:122] "RemoveContainer" containerID="348d499c0112424af3f34d6800c4b29c68b4c09177e08de96541bf3d97ef60e1"
Nov 02 09:58:58.252702 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.252688    4614 scope.go:122] "RemoveContainer" containerID="348d499c0112424af3f34d6800c4b29c68b4c09177e08de96541bf3d97ef60e1"
Nov 02 09:58:58.252961 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:58:58.252942    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348d499c0112424af3f34d6800c4b29c68b4c09177e08de96541bf3d97ef60e1\": container with ID starting with 348d499c0112424af3f34d6800c4b29c68b4c09177e08de96541bf3d97ef60e1 not found: ID does not exist" containerID="348d499c0112424af3f34d6800c4b29c68b4c09177e08de96541bf3d97ef60e1"
Nov 02 09:58:58.253025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.252965    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348d499c0112424af3f34d6800c4b29c68b4c09177e08de96541bf3d97ef60e1"} err="failed to get container status \"348d499c0112424af3f34d6800c4b29c68b4c09177e08de96541bf3d97ef60e1\": rpc error: code = NotFound desc = could not find container \"348d499c0112424af3f34d6800c4b29c68b4c09177e08de96541bf3d97ef60e1\": container with ID starting with 348d499c0112424af3f34d6800c4b29c68b4c09177e08de96541bf3d97ef60e1 not found: ID does not exist"
Nov 02 09:58:58.253025 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.252980    4614 scope.go:122] "RemoveContainer" containerID="c84d182b1e2f5544e44f0607d6fe3eaa3a6bc4ed3b835b78c33460d15ba3f844"
Nov 02 09:58:58.260788 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.260772    4614 scope.go:122] "RemoveContainer" containerID="c84d182b1e2f5544e44f0607d6fe3eaa3a6bc4ed3b835b78c33460d15ba3f844"
Nov 02 09:58:58.261033 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:58:58.261014    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c84d182b1e2f5544e44f0607d6fe3eaa3a6bc4ed3b835b78c33460d15ba3f844\": container with ID starting with c84d182b1e2f5544e44f0607d6fe3eaa3a6bc4ed3b835b78c33460d15ba3f844 not found: ID does not exist" containerID="c84d182b1e2f5544e44f0607d6fe3eaa3a6bc4ed3b835b78c33460d15ba3f844"
Nov 02 09:58:58.261081 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.261038    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c84d182b1e2f5544e44f0607d6fe3eaa3a6bc4ed3b835b78c33460d15ba3f844"} err="failed to get container status \"c84d182b1e2f5544e44f0607d6fe3eaa3a6bc4ed3b835b78c33460d15ba3f844\": rpc error: code = NotFound desc = could not find container \"c84d182b1e2f5544e44f0607d6fe3eaa3a6bc4ed3b835b78c33460d15ba3f844\": container with ID starting with c84d182b1e2f5544e44f0607d6fe3eaa3a6bc4ed3b835b78c33460d15ba3f844 not found: ID does not exist"
Nov 02 09:58:58.264816 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.264793    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"]
Nov 02 09:58:58.267792 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.267773    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-x9j4q"]
Nov 02 09:58:58.282015 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.281993    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"]
Nov 02 09:58:58.288992 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.288972    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-2cjnv"]
Nov 02 09:58:58.305821 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.305796    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"]
Nov 02 09:58:58.312125 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.312094    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9e7156-openshift-default-6bc7994879-cwzxw"]
Nov 02 09:58:58.315432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.315408    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.315432 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.315426    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.315546 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.315437    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ktc9\" (UniqueName: \"kubernetes.io/projected/7a55948d-7d73-4560-85fa-88cb221e3e9d-kube-api-access-5ktc9\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:58.315546 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:58.315447    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a55948d-7d73-4560-85fa-88cb221e3e9d-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 09:58:59.315572 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:59.315544    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a55948d-7d73-4560-85fa-88cb221e3e9d" path="/var/lib/kubelet/pods/7a55948d-7d73-4560-85fa-88cb221e3e9d/volumes"
Nov 02 09:58:59.316026 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:59.316012    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94175a6-949c-4995-b110-b9749c2920b6" path="/var/lib/kubelet/pods/c94175a6-949c-4995-b110-b9749c2920b6/volumes"
Nov 02 09:58:59.316441 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:58:59.316427    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e67b5ee6-666e-4be6-ad46-b0020f405e7e" path="/var/lib/kubelet/pods/e67b5ee6-666e-4be6-ad46-b0020f405e7e/volumes"
Nov 02 09:59:09.027971 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:59:09.027939    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"]
Nov 02 09:59:09.033552 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:59:09.033527    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccef70b8-lq95b"]
Nov 02 09:59:09.315529 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:59:09.315450    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="224ccb7a-946a-4bf6-90a0-4cf63b62be04" path="/var/lib/kubelet/pods/224ccb7a-946a-4bf6-90a0-4cf63b62be04/volumes"
Nov 02 09:59:09.638797 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:59:09.638707    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 09:59:26.031769 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:59:26.031736    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"]
Nov 02 09:59:26.035746 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:59:26.035717    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4910eac-f5wzv"]
Nov 02 09:59:27.316105 ip-10-0-134-174 kubenswrapper[4614]: I1102 09:59:27.316072    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2323d4e8-45f0-4109-9c14-48293a0805d4" path="/var/lib/kubelet/pods/2323d4e8-45f0-4109-9c14-48293a0805d4/volumes"
Nov 02 09:59:39.637807 ip-10-0-134-174 kubenswrapper[4614]: E1102 09:59:39.637761    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:00:00.171641 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.171562    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"]
Nov 02 10:00:00.172242 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172222    4614 cpu_manager.go:414] "containerMap: removing container" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerName="istio-proxy"
Nov 02 10:00:00.172337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172244    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerName="istio-proxy"
Nov 02 10:00:00.172337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172257    4614 cpu_manager.go:414] "containerMap: removing container" podUID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" containerName="istio-proxy"
Nov 02 10:00:00.172337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172266    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" containerName="istio-proxy"
Nov 02 10:00:00.172337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172280    4614 cpu_manager.go:414] "containerMap: removing container" podUID="f58af013-7d2b-4a6b-a55f-e056148cd172" containerName="istio-proxy"
Nov 02 10:00:00.172337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172287    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58af013-7d2b-4a6b-a55f-e056148cd172" containerName="istio-proxy"
Nov 02 10:00:00.172337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172307    4614 cpu_manager.go:414] "containerMap: removing container" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main"
Nov 02 10:00:00.172337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172314    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main"
Nov 02 10:00:00.172337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172325    4614 cpu_manager.go:414] "containerMap: removing container" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy"
Nov 02 10:00:00.172337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172334    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172368    4614 cpu_manager.go:414] "containerMap: removing container" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172377    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172386    4614 cpu_manager.go:414] "containerMap: removing container" podUID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172394    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172420    4614 cpu_manager.go:414] "containerMap: removing container" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172427    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172437    4614 cpu_manager.go:414] "containerMap: removing container" podUID="48b039b3-0a79-43d3-bd83-347582902302" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172445    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b039b3-0a79-43d3-bd83-347582902302" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172461    4614 cpu_manager.go:414] "containerMap: removing container" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172470    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172491    4614 cpu_manager.go:414] "containerMap: removing container" podUID="e67b5ee6-666e-4be6-ad46-b0020f405e7e" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172499    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e67b5ee6-666e-4be6-ad46-b0020f405e7e" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172511    4614 cpu_manager.go:414] "containerMap: removing container" podUID="b4c19aac-3aca-42e1-bd87-22c2a3a3509b" containerName="main"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172518    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c19aac-3aca-42e1-bd87-22c2a3a3509b" containerName="main"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172528    4614 cpu_manager.go:414] "containerMap: removing container" podUID="c94175a6-949c-4995-b110-b9749c2920b6" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172535    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94175a6-949c-4995-b110-b9749c2920b6" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172544    4614 cpu_manager.go:414] "containerMap: removing container" podUID="1137058f-bb14-42e6-ac55-d2104536ffcc" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172551    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1137058f-bb14-42e6-ac55-d2104536ffcc" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172562    4614 cpu_manager.go:414] "containerMap: removing container" podUID="7a55948d-7d73-4560-85fa-88cb221e3e9d" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172569    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a55948d-7d73-4560-85fa-88cb221e3e9d" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172684    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="e67b5ee6-666e-4be6-ad46-b0020f405e7e" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172696    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c19aac-3aca-42e1-bd87-22c2a3a3509b" containerName="main"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172707    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94175a6-949c-4995-b110-b9749c2920b6" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172714    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a55948d-7d73-4560-85fa-88cb221e3e9d" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172725    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c147ae-1d41-49a8-89f9-adf158a7ae5c" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172734    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="858c3cf7-f3f4-41cf-a30e-d8dc86ed0ca6" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172743    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58af013-7d2b-4a6b-a55f-e056148cd172" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172754    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="19dca143-7178-44e5-9e3e-593f99e267c9" containerName="main"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172764    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="839be18b-58e9-404a-8dfa-3a85e612879d" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172772    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="86af445e-d01b-4537-b31a-1a78b30099c9" containerName="main"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172780    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="6421f1c9-7fdd-4cb1-8313-6c9191cb69f7" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172790    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="b807678e-0dc1-4d67-9ead-83414e09f53d" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172802    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b039b3-0a79-43d3-bd83-347582902302" containerName="istio-proxy"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172812    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9eb91a-cdf9-49bf-8b73-329fe059bc42" containerName="main"
Nov 02 10:00:00.172831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.172821    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="1137058f-bb14-42e6-ac55-d2104536ffcc" containerName="istio-proxy"
Nov 02 10:00:00.176772 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.176750    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"]
Nov 02 10:00:00.176936 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.176897    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"
Nov 02 10:00:00.180911 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.180893    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"
Nov 02 10:00:00.188478 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.187698    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"]
Nov 02 10:00:00.189132 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.188981    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 10:00:00.192926 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.192901    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"]
Nov 02 10:00:00.302691 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.302652    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332aba4a-8987-44fc-bf4c-df361f601a20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf\" (UID: \"332aba4a-8987-44fc-bf4c-df361f601a20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"
Nov 02 10:00:00.302910 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.302701    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjx9w\" (UniqueName: \"kubernetes.io/projected/332aba4a-8987-44fc-bf4c-df361f601a20-kube-api-access-hjx9w\") pod \"maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf\" (UID: \"332aba4a-8987-44fc-bf4c-df361f601a20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"
Nov 02 10:00:00.302910 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.302835    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn95t\" (UniqueName: \"kubernetes.io/projected/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e-kube-api-access-hn95t\") pod \"maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx\" (UID: \"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"
Nov 02 10:00:00.303046 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.302988    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx\" (UID: \"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"
Nov 02 10:00:00.404509 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.404474    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn95t\" (UniqueName: \"kubernetes.io/projected/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e-kube-api-access-hn95t\") pod \"maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx\" (UID: \"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"
Nov 02 10:00:00.404717 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.404553    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx\" (UID: \"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"
Nov 02 10:00:00.404717 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.404597    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332aba4a-8987-44fc-bf4c-df361f601a20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf\" (UID: \"332aba4a-8987-44fc-bf4c-df361f601a20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"
Nov 02 10:00:00.404717 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.404617    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjx9w\" (UniqueName: \"kubernetes.io/projected/332aba4a-8987-44fc-bf4c-df361f601a20-kube-api-access-hjx9w\") pod \"maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf\" (UID: \"332aba4a-8987-44fc-bf4c-df361f601a20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"
Nov 02 10:00:00.405250 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.405228    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332aba4a-8987-44fc-bf4c-df361f601a20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf\" (UID: \"332aba4a-8987-44fc-bf4c-df361f601a20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"
Nov 02 10:00:00.405291 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.405232    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx\" (UID: \"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"
Nov 02 10:00:00.419296 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.419265    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjx9w\" (UniqueName: \"kubernetes.io/projected/332aba4a-8987-44fc-bf4c-df361f601a20-kube-api-access-hjx9w\") pod \"maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf\" (UID: \"332aba4a-8987-44fc-bf4c-df361f601a20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"
Nov 02 10:00:00.419791 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.419773    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn95t\" (UniqueName: \"kubernetes.io/projected/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e-kube-api-access-hn95t\") pod \"maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx\" (UID: \"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"
Nov 02 10:00:00.498935 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.498900    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"
Nov 02 10:00:00.508230 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.508204    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"
Nov 02 10:00:00.784199 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.784166    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"]
Nov 02 10:00:00.786723 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:00:00.786693    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87e3019_69d9_4cbb_9c6f_5fe5bbec2d6e.slice/crio-7101594b062955540b6bd2ce0d21807e207cc05fcf8a49226369753e785b8902 WatchSource:0}: Error finding container 7101594b062955540b6bd2ce0d21807e207cc05fcf8a49226369753e785b8902: Status 404 returned error can't find the container with id 7101594b062955540b6bd2ce0d21807e207cc05fcf8a49226369753e785b8902
Nov 02 10:00:00.796721 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:00.796698    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"]
Nov 02 10:00:00.799019 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:00:00.798995    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332aba4a_8987_44fc_bf4c_df361f601a20.slice/crio-f6c99c4f26c061d8518ab942234f2b29d738e10b09814184134860a328d23462 WatchSource:0}: Error finding container f6c99c4f26c061d8518ab942234f2b29d738e10b09814184134860a328d23462: Status 404 returned error can't find the container with id f6c99c4f26c061d8518ab942234f2b29d738e10b09814184134860a328d23462
Nov 02 10:00:01.650064 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:01.650028    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf" event={"ID":"332aba4a-8987-44fc-bf4c-df361f601a20","Type":"ContainerStarted","Data":"7e99193f8406f2d0275668471406886d0499f41914aad1b54399fe6cb0524c72"}
Nov 02 10:00:01.650531 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:01.650100    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf" event={"ID":"332aba4a-8987-44fc-bf4c-df361f601a20","Type":"ContainerStarted","Data":"f6c99c4f26c061d8518ab942234f2b29d738e10b09814184134860a328d23462"}
Nov 02 10:00:01.651808 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:01.651782    4614 generic.go:356] "Generic (PLEG): container finished" podID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" containerID="39778af49c00f47bd74875071e8719ceca49fbb7588862e5e8b52253991400c9" exitCode=60
Nov 02 10:00:01.651913 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:01.651844    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx" event={"ID":"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e","Type":"ContainerDied","Data":"39778af49c00f47bd74875071e8719ceca49fbb7588862e5e8b52253991400c9"}
Nov 02 10:00:01.651913 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:01.651876    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx" event={"ID":"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e","Type":"ContainerStarted","Data":"7101594b062955540b6bd2ce0d21807e207cc05fcf8a49226369753e785b8902"}
Nov 02 10:00:01.652065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:01.652046    4614 scope.go:122] "RemoveContainer" containerID="39778af49c00f47bd74875071e8719ceca49fbb7588862e5e8b52253991400c9"
Nov 02 10:00:01.669250 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:01.669207    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf" podStartSLOduration=1.669194907 podStartE2EDuration="1.669194907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:00:01.668041373 +0000 UTC m=+1482.500699479" watchObservedRunningTime="2026-11-02 10:00:01.669194907 +0000 UTC m=+1482.501852988"
Nov 02 10:00:02.659121 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:02.659086    4614 generic.go:356] "Generic (PLEG): container finished" podID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" containerID="e2417d949b5144a9c83ed612e2a6ca0417c56392c2c8d1e7c6b21192825713c9" exitCode=60
Nov 02 10:00:02.659554 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:02.659171    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx" event={"ID":"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e","Type":"ContainerDied","Data":"e2417d949b5144a9c83ed612e2a6ca0417c56392c2c8d1e7c6b21192825713c9"}
Nov 02 10:00:02.659554 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:02.659231    4614 scope.go:122] "RemoveContainer" containerID="39778af49c00f47bd74875071e8719ceca49fbb7588862e5e8b52253991400c9"
Nov 02 10:00:02.659554 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:02.659515    4614 scope.go:122] "RemoveContainer" containerID="e2417d949b5144a9c83ed612e2a6ca0417c56392c2c8d1e7c6b21192825713c9"
Nov 02 10:00:02.659819 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:02.659795    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx_odh-ai-gateway-infra(c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx" podUID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e"
Nov 02 10:00:02.660994 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:02.660964    4614 generic.go:356] "Generic (PLEG): container finished" podID="332aba4a-8987-44fc-bf4c-df361f601a20" containerID="7e99193f8406f2d0275668471406886d0499f41914aad1b54399fe6cb0524c72" exitCode=60
Nov 02 10:00:02.661086 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:02.661006    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf" event={"ID":"332aba4a-8987-44fc-bf4c-df361f601a20","Type":"ContainerDied","Data":"7e99193f8406f2d0275668471406886d0499f41914aad1b54399fe6cb0524c72"}
Nov 02 10:00:02.661248 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:02.661235    4614 scope.go:122] "RemoveContainer" containerID="7e99193f8406f2d0275668471406886d0499f41914aad1b54399fe6cb0524c72"
Nov 02 10:00:03.669009 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:03.668985    4614 scope.go:122] "RemoveContainer" containerID="e2417d949b5144a9c83ed612e2a6ca0417c56392c2c8d1e7c6b21192825713c9"
Nov 02 10:00:03.669582 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:03.669243    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx_odh-ai-gateway-infra(c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx" podUID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e"
Nov 02 10:00:03.670729 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:03.670703    4614 generic.go:356] "Generic (PLEG): container finished" podID="332aba4a-8987-44fc-bf4c-df361f601a20" containerID="217755abe99097f0533462f3453b26b638e29eb90db44c7baaf84570f9bebb56" exitCode=60
Nov 02 10:00:03.670856 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:03.670795    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf" event={"ID":"332aba4a-8987-44fc-bf4c-df361f601a20","Type":"ContainerDied","Data":"217755abe99097f0533462f3453b26b638e29eb90db44c7baaf84570f9bebb56"}
Nov 02 10:00:03.670942 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:03.670857    4614 scope.go:122] "RemoveContainer" containerID="7e99193f8406f2d0275668471406886d0499f41914aad1b54399fe6cb0524c72"
Nov 02 10:00:03.671051 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:03.671029    4614 scope.go:122] "RemoveContainer" containerID="217755abe99097f0533462f3453b26b638e29eb90db44c7baaf84570f9bebb56"
Nov 02 10:00:03.671294 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:03.671273    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf_odh-ai-gateway-infra(332aba4a-8987-44fc-bf4c-df361f601a20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf" podUID="332aba4a-8987-44fc-bf4c-df361f601a20"
Nov 02 10:00:04.071803 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:04.071773    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"]
Nov 02 10:00:04.071982 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:04.071963    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm" podUID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerName="istio-proxy" containerID="cri-o://033e00d3038edac90628ea16925637a39ff4a0d03aa5cd2ece559f549b7b9e21" gracePeriod=30
Nov 02 10:00:04.078407 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:04.078380    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"]
Nov 02 10:00:04.078617 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:04.078589    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q" podUID="6f3a1400-7110-4d96-884a-f3874bf496ce" containerName="istio-proxy" containerID="cri-o://89ae424a5385219343666279fae08b0378832384d900464abc65d11ff8e29f85" gracePeriod=30
Nov 02 10:00:04.080259 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:04.080210    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"]
Nov 02 10:00:04.080490 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:04.080464    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg" podUID="adde7f6c-619c-4d32-9d5e-0540c56c96fc" containerName="istio-proxy" containerID="cri-o://3cc6d4cb080599d75446c77f1d0d3314d926b0eaa40e182e42d03481f4e69424" gracePeriod=30
Nov 02 10:00:04.082632 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:04.082610    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"]
Nov 02 10:00:04.082818 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:04.082792    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl" podUID="125f651d-4e3c-4f0e-8748-50e55a48407d" containerName="istio-proxy" containerID="cri-o://8cfbc7c79cdaa827a0fa5bce605837560935f7a6bdfcdaa4b00e087aa22bd795" gracePeriod=30
Nov 02 10:00:04.679043 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:04.679015    4614 scope.go:122] "RemoveContainer" containerID="217755abe99097f0533462f3453b26b638e29eb90db44c7baaf84570f9bebb56"
Nov 02 10:00:04.679502 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:04.679340    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf_odh-ai-gateway-infra(332aba4a-8987-44fc-bf4c-df361f601a20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf" podUID="332aba4a-8987-44fc-bf4c-df361f601a20"
Nov 02 10:00:09.369384 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.368751    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 10:00:09.409799 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.409762    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 10:00:09.436666 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.436620    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 10:00:09.444510 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.444482    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 10:00:09.459091 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.459060    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-token\" (UniqueName: \"kubernetes.io/projected/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-token\") pod \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") "
Nov 02 10:00:09.459225 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.459173    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-credential-socket\") pod \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") "
Nov 02 10:00:09.459225 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.459220    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-podinfo\") pod \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") "
Nov 02 10:00:09.459408 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.459389    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a1b0ce-e12b-4227-90be-f17a736ab20d-kube-api-access-g7gvv\" (UniqueName: \"kubernetes.io/projected/a0a1b0ce-e12b-4227-90be-f17a736ab20d-kube-api-access-g7gvv\") pod \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") "
Nov 02 10:00:09.459476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.459438    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-workload-socket\") pod \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") "
Nov 02 10:00:09.459530 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.459485    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-workload-certs\") pod \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") "
Nov 02 10:00:09.459530 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.459516    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istiod-ca-cert\") pod \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") "
Nov 02 10:00:09.459657 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.459531    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-credential-socket" pod "a0a1b0ce-e12b-4227-90be-f17a736ab20d" (UID: "a0a1b0ce-e12b-4227-90be-f17a736ab20d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.459657 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.459547    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-data\") pod \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") "
Nov 02 10:00:09.459657 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.459583    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-envoy\") pod \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\" (UID: \"a0a1b0ce-e12b-4227-90be-f17a736ab20d\") "
Nov 02 10:00:09.459815 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.459756    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-workload-socket" pod "a0a1b0ce-e12b-4227-90be-f17a736ab20d" (UID: "a0a1b0ce-e12b-4227-90be-f17a736ab20d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.459974 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.459946    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-data" pod "a0a1b0ce-e12b-4227-90be-f17a736ab20d" (UID: "a0a1b0ce-e12b-4227-90be-f17a736ab20d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.460098 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.459967    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-workload-certs" pod "a0a1b0ce-e12b-4227-90be-f17a736ab20d" (UID: "a0a1b0ce-e12b-4227-90be-f17a736ab20d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.460241 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.459969    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istiod-ca-cert" pod "a0a1b0ce-e12b-4227-90be-f17a736ab20d" (UID: "a0a1b0ce-e12b-4227-90be-f17a736ab20d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:00:09.461337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.461312    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.461469 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.461344    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.461469 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.461406    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.461469 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.461422    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.461469 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.461436    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.463123 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.463087    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-envoy" pod "a0a1b0ce-e12b-4227-90be-f17a736ab20d" (UID: "a0a1b0ce-e12b-4227-90be-f17a736ab20d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.464248 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.464223    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-podinfo" pod "a0a1b0ce-e12b-4227-90be-f17a736ab20d" (UID: "a0a1b0ce-e12b-4227-90be-f17a736ab20d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:00:09.465196 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.465155    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-token" pod "a0a1b0ce-e12b-4227-90be-f17a736ab20d" (UID: "a0a1b0ce-e12b-4227-90be-f17a736ab20d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:09.466800 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.466777    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a1b0ce-e12b-4227-90be-f17a736ab20d-kube-api-access-g7gvv" pod "a0a1b0ce-e12b-4227-90be-f17a736ab20d" (UID: "a0a1b0ce-e12b-4227-90be-f17a736ab20d"). InnerVolumeSpecName "kube-api-access-g7gvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:09.562057 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562019    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-envoy\") pod \"6f3a1400-7110-4d96-884a-f3874bf496ce\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") "
Nov 02 10:00:09.562057 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562054    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-podinfo\") pod \"6f3a1400-7110-4d96-884a-f3874bf496ce\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") "
Nov 02 10:00:09.562308 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562077    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-data\") pod \"6f3a1400-7110-4d96-884a-f3874bf496ce\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") "
Nov 02 10:00:09.562308 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562115    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f3a1400-7110-4d96-884a-f3874bf496ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3a1400-7110-4d96-884a-f3874bf496ce-istiod-ca-cert\") pod \"6f3a1400-7110-4d96-884a-f3874bf496ce\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") "
Nov 02 10:00:09.562308 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562155    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adde7f6c-619c-4d32-9d5e-0540c56c96fc-kube-api-access-52rr4\" (UniqueName: \"kubernetes.io/projected/adde7f6c-619c-4d32-9d5e-0540c56c96fc-kube-api-access-52rr4\") pod \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") "
Nov 02 10:00:09.562308 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562174    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-data\") pod \"125f651d-4e3c-4f0e-8748-50e55a48407d\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") "
Nov 02 10:00:09.562308 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562192    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-workload-socket\") pod \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") "
Nov 02 10:00:09.562308 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562216    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-workload-certs\") pod \"125f651d-4e3c-4f0e-8748-50e55a48407d\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") "
Nov 02 10:00:09.562308 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562242    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-envoy\") pod \"125f651d-4e3c-4f0e-8748-50e55a48407d\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") "
Nov 02 10:00:09.562308 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562281    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-token\" (UniqueName: \"kubernetes.io/projected/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-token\") pod \"125f651d-4e3c-4f0e-8748-50e55a48407d\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") "
Nov 02 10:00:09.562775 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562319    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/125f651d-4e3c-4f0e-8748-50e55a48407d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/125f651d-4e3c-4f0e-8748-50e55a48407d-istiod-ca-cert\") pod \"125f651d-4e3c-4f0e-8748-50e55a48407d\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") "
Nov 02 10:00:09.562775 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562363    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-token\" (UniqueName: \"kubernetes.io/projected/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-token\") pod \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") "
Nov 02 10:00:09.562775 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562411    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-workload-socket\") pod \"6f3a1400-7110-4d96-884a-f3874bf496ce\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") "
Nov 02 10:00:09.562775 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562451    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-credential-socket\") pod \"125f651d-4e3c-4f0e-8748-50e55a48407d\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") "
Nov 02 10:00:09.562775 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562491    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-credential-socket\") pod \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") "
Nov 02 10:00:09.562775 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562549    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-podinfo\") pod \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") "
Nov 02 10:00:09.562775 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562579    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-credential-socket\") pod \"6f3a1400-7110-4d96-884a-f3874bf496ce\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") "
Nov 02 10:00:09.562775 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562605    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-token\" (UniqueName: \"kubernetes.io/projected/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-token\") pod \"6f3a1400-7110-4d96-884a-f3874bf496ce\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") "
Nov 02 10:00:09.562775 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562649    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-workload-socket\") pod \"125f651d-4e3c-4f0e-8748-50e55a48407d\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") "
Nov 02 10:00:09.562775 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562703    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-podinfo\") pod \"125f651d-4e3c-4f0e-8748-50e55a48407d\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") "
Nov 02 10:00:09.562775 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562753    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-data\") pod \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") "
Nov 02 10:00:09.563369 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562788    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-workload-certs\") pod \"6f3a1400-7110-4d96-884a-f3874bf496ce\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") "
Nov 02 10:00:09.563369 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562859    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istiod-ca-cert\") pod \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") "
Nov 02 10:00:09.563369 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562912    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f3a1400-7110-4d96-884a-f3874bf496ce-kube-api-access-5jccn\" (UniqueName: \"kubernetes.io/projected/6f3a1400-7110-4d96-884a-f3874bf496ce-kube-api-access-5jccn\") pod \"6f3a1400-7110-4d96-884a-f3874bf496ce\" (UID: \"6f3a1400-7110-4d96-884a-f3874bf496ce\") "
Nov 02 10:00:09.563369 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562949    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125f651d-4e3c-4f0e-8748-50e55a48407d-kube-api-access-r7bwg\" (UniqueName: \"kubernetes.io/projected/125f651d-4e3c-4f0e-8748-50e55a48407d-kube-api-access-r7bwg\") pod \"125f651d-4e3c-4f0e-8748-50e55a48407d\" (UID: \"125f651d-4e3c-4f0e-8748-50e55a48407d\") "
Nov 02 10:00:09.563369 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.563001    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-workload-certs\") pod \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") "
Nov 02 10:00:09.563369 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.563030    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-envoy\") pod \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\" (UID: \"adde7f6c-619c-4d32-9d5e-0540c56c96fc\") "
Nov 02 10:00:09.563714 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.563594    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.563714 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.563618    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.563714 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.563635    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a1b0ce-e12b-4227-90be-f17a736ab20d-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.563714 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.563654    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7gvv\" (UniqueName: \"kubernetes.io/projected/a0a1b0ce-e12b-4227-90be-f17a736ab20d-kube-api-access-g7gvv\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.563928 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562486    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-data" pod "125f651d-4e3c-4f0e-8748-50e55a48407d" (UID: "125f651d-4e3c-4f0e-8748-50e55a48407d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.563928 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562504    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-workload-socket" pod "adde7f6c-619c-4d32-9d5e-0540c56c96fc" (UID: "adde7f6c-619c-4d32-9d5e-0540c56c96fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.563928 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562731    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-workload-socket" pod "6f3a1400-7110-4d96-884a-f3874bf496ce" (UID: "6f3a1400-7110-4d96-884a-f3874bf496ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.563928 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562749    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-data" pod "6f3a1400-7110-4d96-884a-f3874bf496ce" (UID: "6f3a1400-7110-4d96-884a-f3874bf496ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.564137 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562880    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125f651d-4e3c-4f0e-8748-50e55a48407d-istiod-ca-cert" pod "125f651d-4e3c-4f0e-8748-50e55a48407d" (UID: "125f651d-4e3c-4f0e-8748-50e55a48407d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:00:09.564137 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.562942    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f3a1400-7110-4d96-884a-f3874bf496ce-istiod-ca-cert" pod "6f3a1400-7110-4d96-884a-f3874bf496ce" (UID: "6f3a1400-7110-4d96-884a-f3874bf496ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:00:09.564137 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.563314    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-workload-certs" pod "125f651d-4e3c-4f0e-8748-50e55a48407d" (UID: "125f651d-4e3c-4f0e-8748-50e55a48407d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.564137 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.563512    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-workload-socket" pod "125f651d-4e3c-4f0e-8748-50e55a48407d" (UID: "125f651d-4e3c-4f0e-8748-50e55a48407d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.564137 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.563668    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-credential-socket" pod "125f651d-4e3c-4f0e-8748-50e55a48407d" (UID: "125f651d-4e3c-4f0e-8748-50e55a48407d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.564137 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.563881    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-credential-socket" pod "adde7f6c-619c-4d32-9d5e-0540c56c96fc" (UID: "adde7f6c-619c-4d32-9d5e-0540c56c96fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.565394 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.564827    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adde7f6c-619c-4d32-9d5e-0540c56c96fc-kube-api-access-52rr4" pod "adde7f6c-619c-4d32-9d5e-0540c56c96fc" (UID: "adde7f6c-619c-4d32-9d5e-0540c56c96fc"). InnerVolumeSpecName "kube-api-access-52rr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:09.565394 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.565051    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-credential-socket" pod "6f3a1400-7110-4d96-884a-f3874bf496ce" (UID: "6f3a1400-7110-4d96-884a-f3874bf496ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.565394 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.565105    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-envoy" pod "125f651d-4e3c-4f0e-8748-50e55a48407d" (UID: "125f651d-4e3c-4f0e-8748-50e55a48407d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.565394 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.565104    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-envoy" pod "6f3a1400-7110-4d96-884a-f3874bf496ce" (UID: "6f3a1400-7110-4d96-884a-f3874bf496ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.565394 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.565199    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-token" pod "adde7f6c-619c-4d32-9d5e-0540c56c96fc" (UID: "adde7f6c-619c-4d32-9d5e-0540c56c96fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:09.565394 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.565290    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istiod-ca-cert" pod "adde7f6c-619c-4d32-9d5e-0540c56c96fc" (UID: "adde7f6c-619c-4d32-9d5e-0540c56c96fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:00:09.565394 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.565332    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-data" pod "adde7f6c-619c-4d32-9d5e-0540c56c96fc" (UID: "adde7f6c-619c-4d32-9d5e-0540c56c96fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.565835 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.565580    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-workload-certs" pod "6f3a1400-7110-4d96-884a-f3874bf496ce" (UID: "6f3a1400-7110-4d96-884a-f3874bf496ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.565835 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.565775    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-podinfo" pod "6f3a1400-7110-4d96-884a-f3874bf496ce" (UID: "6f3a1400-7110-4d96-884a-f3874bf496ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:00:09.565835 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.565799    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-workload-certs" pod "adde7f6c-619c-4d32-9d5e-0540c56c96fc" (UID: "adde7f6c-619c-4d32-9d5e-0540c56c96fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.565998 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.565863    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-token" pod "125f651d-4e3c-4f0e-8748-50e55a48407d" (UID: "125f651d-4e3c-4f0e-8748-50e55a48407d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:09.566443 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.566407    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-envoy" pod "adde7f6c-619c-4d32-9d5e-0540c56c96fc" (UID: "adde7f6c-619c-4d32-9d5e-0540c56c96fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:00:09.566551 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.566488    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-podinfo" pod "adde7f6c-619c-4d32-9d5e-0540c56c96fc" (UID: "adde7f6c-619c-4d32-9d5e-0540c56c96fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:00:09.567386 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.567342    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-podinfo" pod "125f651d-4e3c-4f0e-8748-50e55a48407d" (UID: "125f651d-4e3c-4f0e-8748-50e55a48407d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:00:09.567514 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.567492    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-token" pod "6f3a1400-7110-4d96-884a-f3874bf496ce" (UID: "6f3a1400-7110-4d96-884a-f3874bf496ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:09.567597 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.567581    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125f651d-4e3c-4f0e-8748-50e55a48407d-kube-api-access-r7bwg" pod "125f651d-4e3c-4f0e-8748-50e55a48407d" (UID: "125f651d-4e3c-4f0e-8748-50e55a48407d"). InnerVolumeSpecName "kube-api-access-r7bwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:09.567644 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.567631    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3a1400-7110-4d96-884a-f3874bf496ce-kube-api-access-5jccn" pod "6f3a1400-7110-4d96-884a-f3874bf496ce" (UID: "6f3a1400-7110-4d96-884a-f3874bf496ce"). InnerVolumeSpecName "kube-api-access-5jccn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:09.636960 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:09.636865    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:00:09.664707 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664678    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664707 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664707    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664722    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664736    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jccn\" (UniqueName: \"kubernetes.io/projected/6f3a1400-7110-4d96-884a-f3874bf496ce-kube-api-access-5jccn\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664750    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7bwg\" (UniqueName: \"kubernetes.io/projected/125f651d-4e3c-4f0e-8748-50e55a48407d-kube-api-access-r7bwg\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664761    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664773    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664786    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664801    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664813    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664827    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f3a1400-7110-4d96-884a-f3874bf496ce-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664841    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52rr4\" (UniqueName: \"kubernetes.io/projected/adde7f6c-619c-4d32-9d5e-0540c56c96fc-kube-api-access-52rr4\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664854    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664866    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664878    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664890    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664903    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664916    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/125f651d-4e3c-4f0e-8748-50e55a48407d-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664928    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664942    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664955    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664969    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adde7f6c-619c-4d32-9d5e-0540c56c96fc-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664982    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adde7f6c-619c-4d32-9d5e-0540c56c96fc-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.664986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.664996    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f3a1400-7110-4d96-884a-f3874bf496ce-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.665687 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.665009    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f3a1400-7110-4d96-884a-f3874bf496ce-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.665687 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.665022    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/125f651d-4e3c-4f0e-8748-50e55a48407d-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.665687 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.665038    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/125f651d-4e3c-4f0e-8748-50e55a48407d-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:09.708821 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.708791    4614 generic.go:356] "Generic (PLEG): container finished" podID="adde7f6c-619c-4d32-9d5e-0540c56c96fc" containerID="3cc6d4cb080599d75446c77f1d0d3314d926b0eaa40e182e42d03481f4e69424" exitCode=0
Nov 02 10:00:09.709010 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.708837    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"
Nov 02 10:00:09.709010 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.708874    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg" event={"ID":"adde7f6c-619c-4d32-9d5e-0540c56c96fc","Type":"ContainerDied","Data":"3cc6d4cb080599d75446c77f1d0d3314d926b0eaa40e182e42d03481f4e69424"}
Nov 02 10:00:09.709010 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.708945    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg" event={"ID":"adde7f6c-619c-4d32-9d5e-0540c56c96fc","Type":"ContainerDied","Data":"54b3c1aff9cc6ba056ad33ec3272532e6d63ac1cff4595950b7c1b366b8b9c62"}
Nov 02 10:00:09.709010 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.708975    4614 scope.go:122] "RemoveContainer" containerID="3cc6d4cb080599d75446c77f1d0d3314d926b0eaa40e182e42d03481f4e69424"
Nov 02 10:00:09.710416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.710388    4614 generic.go:356] "Generic (PLEG): container finished" podID="6f3a1400-7110-4d96-884a-f3874bf496ce" containerID="89ae424a5385219343666279fae08b0378832384d900464abc65d11ff8e29f85" exitCode=0
Nov 02 10:00:09.710531 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.710437    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"
Nov 02 10:00:09.710531 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.710482    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q" event={"ID":"6f3a1400-7110-4d96-884a-f3874bf496ce","Type":"ContainerDied","Data":"89ae424a5385219343666279fae08b0378832384d900464abc65d11ff8e29f85"}
Nov 02 10:00:09.710658 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.710540    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q" event={"ID":"6f3a1400-7110-4d96-884a-f3874bf496ce","Type":"ContainerDied","Data":"eec9bd5255ad33a4e79e8a4f80965cdb18b0a3f35dd83b5213be134f85102348"}
Nov 02 10:00:09.712492 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.712466    4614 generic.go:356] "Generic (PLEG): container finished" podID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerID="033e00d3038edac90628ea16925637a39ff4a0d03aa5cd2ece559f549b7b9e21" exitCode=0
Nov 02 10:00:09.712599 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.712496    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm" event={"ID":"a0a1b0ce-e12b-4227-90be-f17a736ab20d","Type":"ContainerDied","Data":"033e00d3038edac90628ea16925637a39ff4a0d03aa5cd2ece559f549b7b9e21"}
Nov 02 10:00:09.712599 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.712516    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"
Nov 02 10:00:09.712599 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.712545    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm" event={"ID":"a0a1b0ce-e12b-4227-90be-f17a736ab20d","Type":"ContainerDied","Data":"1dc40291cc5d6692dd3327c00870398e510895cfce4ab0ae666678a60095b11e"}
Nov 02 10:00:09.714089 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.714054    4614 generic.go:356] "Generic (PLEG): container finished" podID="125f651d-4e3c-4f0e-8748-50e55a48407d" containerID="8cfbc7c79cdaa827a0fa5bce605837560935f7a6bdfcdaa4b00e087aa22bd795" exitCode=0
Nov 02 10:00:09.714230 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.714107    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl" event={"ID":"125f651d-4e3c-4f0e-8748-50e55a48407d","Type":"ContainerDied","Data":"8cfbc7c79cdaa827a0fa5bce605837560935f7a6bdfcdaa4b00e087aa22bd795"}
Nov 02 10:00:09.714230 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.714148    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl" event={"ID":"125f651d-4e3c-4f0e-8748-50e55a48407d","Type":"ContainerDied","Data":"9a7cec6b20f2bc1e1d35c2553b9783593cd0a733b6fc1905f07d220b2992dd51"}
Nov 02 10:00:09.714427 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.714252    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"
Nov 02 10:00:09.721402 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.721342    4614 scope.go:122] "RemoveContainer" containerID="3cc6d4cb080599d75446c77f1d0d3314d926b0eaa40e182e42d03481f4e69424"
Nov 02 10:00:09.721846 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:09.721708    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc6d4cb080599d75446c77f1d0d3314d926b0eaa40e182e42d03481f4e69424\": container with ID starting with 3cc6d4cb080599d75446c77f1d0d3314d926b0eaa40e182e42d03481f4e69424 not found: ID does not exist" containerID="3cc6d4cb080599d75446c77f1d0d3314d926b0eaa40e182e42d03481f4e69424"
Nov 02 10:00:09.721846 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.721744    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc6d4cb080599d75446c77f1d0d3314d926b0eaa40e182e42d03481f4e69424"} err="failed to get container status \"3cc6d4cb080599d75446c77f1d0d3314d926b0eaa40e182e42d03481f4e69424\": rpc error: code = NotFound desc = could not find container \"3cc6d4cb080599d75446c77f1d0d3314d926b0eaa40e182e42d03481f4e69424\": container with ID starting with 3cc6d4cb080599d75446c77f1d0d3314d926b0eaa40e182e42d03481f4e69424 not found: ID does not exist"
Nov 02 10:00:09.721846 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.721766    4614 scope.go:122] "RemoveContainer" containerID="89ae424a5385219343666279fae08b0378832384d900464abc65d11ff8e29f85"
Nov 02 10:00:09.733436 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.733414    4614 scope.go:122] "RemoveContainer" containerID="89ae424a5385219343666279fae08b0378832384d900464abc65d11ff8e29f85"
Nov 02 10:00:09.733701 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:09.733683    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ae424a5385219343666279fae08b0378832384d900464abc65d11ff8e29f85\": container with ID starting with 89ae424a5385219343666279fae08b0378832384d900464abc65d11ff8e29f85 not found: ID does not exist" containerID="89ae424a5385219343666279fae08b0378832384d900464abc65d11ff8e29f85"
Nov 02 10:00:09.733776 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.733708    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ae424a5385219343666279fae08b0378832384d900464abc65d11ff8e29f85"} err="failed to get container status \"89ae424a5385219343666279fae08b0378832384d900464abc65d11ff8e29f85\": rpc error: code = NotFound desc = could not find container \"89ae424a5385219343666279fae08b0378832384d900464abc65d11ff8e29f85\": container with ID starting with 89ae424a5385219343666279fae08b0378832384d900464abc65d11ff8e29f85 not found: ID does not exist"
Nov 02 10:00:09.733776 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.733726    4614 scope.go:122] "RemoveContainer" containerID="033e00d3038edac90628ea16925637a39ff4a0d03aa5cd2ece559f549b7b9e21"
Nov 02 10:00:09.742115 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.742089    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"]
Nov 02 10:00:09.742494 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.742470    4614 scope.go:122] "RemoveContainer" containerID="f9e7be1164127f4bdff0792e229b1baf8b546dc62c84e4ed84eb70b583652e26"
Nov 02 10:00:09.745089 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.745070    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-fvrvg"]
Nov 02 10:00:09.750932 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.750916    4614 scope.go:122] "RemoveContainer" containerID="033e00d3038edac90628ea16925637a39ff4a0d03aa5cd2ece559f549b7b9e21"
Nov 02 10:00:09.751216 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:09.751195    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033e00d3038edac90628ea16925637a39ff4a0d03aa5cd2ece559f549b7b9e21\": container with ID starting with 033e00d3038edac90628ea16925637a39ff4a0d03aa5cd2ece559f549b7b9e21 not found: ID does not exist" containerID="033e00d3038edac90628ea16925637a39ff4a0d03aa5cd2ece559f549b7b9e21"
Nov 02 10:00:09.751268 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.751222    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033e00d3038edac90628ea16925637a39ff4a0d03aa5cd2ece559f549b7b9e21"} err="failed to get container status \"033e00d3038edac90628ea16925637a39ff4a0d03aa5cd2ece559f549b7b9e21\": rpc error: code = NotFound desc = could not find container \"033e00d3038edac90628ea16925637a39ff4a0d03aa5cd2ece559f549b7b9e21\": container with ID starting with 033e00d3038edac90628ea16925637a39ff4a0d03aa5cd2ece559f549b7b9e21 not found: ID does not exist"
Nov 02 10:00:09.751268 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.751239    4614 scope.go:122] "RemoveContainer" containerID="f9e7be1164127f4bdff0792e229b1baf8b546dc62c84e4ed84eb70b583652e26"
Nov 02 10:00:09.751505 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:09.751487    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e7be1164127f4bdff0792e229b1baf8b546dc62c84e4ed84eb70b583652e26\": container with ID starting with f9e7be1164127f4bdff0792e229b1baf8b546dc62c84e4ed84eb70b583652e26 not found: ID does not exist" containerID="f9e7be1164127f4bdff0792e229b1baf8b546dc62c84e4ed84eb70b583652e26"
Nov 02 10:00:09.751565 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.751513    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e7be1164127f4bdff0792e229b1baf8b546dc62c84e4ed84eb70b583652e26"} err="failed to get container status \"f9e7be1164127f4bdff0792e229b1baf8b546dc62c84e4ed84eb70b583652e26\": rpc error: code = NotFound desc = could not find container \"f9e7be1164127f4bdff0792e229b1baf8b546dc62c84e4ed84eb70b583652e26\": container with ID starting with f9e7be1164127f4bdff0792e229b1baf8b546dc62c84e4ed84eb70b583652e26 not found: ID does not exist"
Nov 02 10:00:09.751565 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.751529    4614 scope.go:122] "RemoveContainer" containerID="8cfbc7c79cdaa827a0fa5bce605837560935f7a6bdfcdaa4b00e087aa22bd795"
Nov 02 10:00:09.758330 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.758310    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"]
Nov 02 10:00:09.762593 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.762570    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-r2twl"]
Nov 02 10:00:09.765831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.765810    4614 scope.go:122] "RemoveContainer" containerID="8cfbc7c79cdaa827a0fa5bce605837560935f7a6bdfcdaa4b00e087aa22bd795"
Nov 02 10:00:09.766195 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:09.766175    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cfbc7c79cdaa827a0fa5bce605837560935f7a6bdfcdaa4b00e087aa22bd795\": container with ID starting with 8cfbc7c79cdaa827a0fa5bce605837560935f7a6bdfcdaa4b00e087aa22bd795 not found: ID does not exist" containerID="8cfbc7c79cdaa827a0fa5bce605837560935f7a6bdfcdaa4b00e087aa22bd795"
Nov 02 10:00:09.766265 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.766202    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cfbc7c79cdaa827a0fa5bce605837560935f7a6bdfcdaa4b00e087aa22bd795"} err="failed to get container status \"8cfbc7c79cdaa827a0fa5bce605837560935f7a6bdfcdaa4b00e087aa22bd795\": rpc error: code = NotFound desc = could not find container \"8cfbc7c79cdaa827a0fa5bce605837560935f7a6bdfcdaa4b00e087aa22bd795\": container with ID starting with 8cfbc7c79cdaa827a0fa5bce605837560935f7a6bdfcdaa4b00e087aa22bd795 not found: ID does not exist"
Nov 02 10:00:09.774800 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.774780    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"]
Nov 02 10:00:09.778934 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.778913    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-4kb6q"]
Nov 02 10:00:09.790112 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.790084    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"]
Nov 02 10:00:09.795161 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:09.795140    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-xl5jm"]
Nov 02 10:00:11.315514 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:11.315480    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125f651d-4e3c-4f0e-8748-50e55a48407d" path="/var/lib/kubelet/pods/125f651d-4e3c-4f0e-8748-50e55a48407d/volumes"
Nov 02 10:00:11.316232 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:11.316184    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f3a1400-7110-4d96-884a-f3874bf496ce" path="/var/lib/kubelet/pods/6f3a1400-7110-4d96-884a-f3874bf496ce/volumes"
Nov 02 10:00:11.317005 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:11.316981    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" path="/var/lib/kubelet/pods/a0a1b0ce-e12b-4227-90be-f17a736ab20d/volumes"
Nov 02 10:00:11.318217 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:11.318196    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adde7f6c-619c-4d32-9d5e-0540c56c96fc" path="/var/lib/kubelet/pods/adde7f6c-619c-4d32-9d5e-0540c56c96fc/volumes"
Nov 02 10:00:14.309259 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:14.309233    4614 scope.go:122] "RemoveContainer" containerID="e2417d949b5144a9c83ed612e2a6ca0417c56392c2c8d1e7c6b21192825713c9"
Nov 02 10:00:15.498528 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.498490    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"]
Nov 02 10:00:15.498951 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.498937    4614 cpu_manager.go:414] "containerMap: removing container" podUID="125f651d-4e3c-4f0e-8748-50e55a48407d" containerName="istio-proxy"
Nov 02 10:00:15.499005 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.498953    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125f651d-4e3c-4f0e-8748-50e55a48407d" containerName="istio-proxy"
Nov 02 10:00:15.499005 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.498970    4614 cpu_manager.go:414] "containerMap: removing container" podUID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerName="istio-proxy"
Nov 02 10:00:15.499005 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.498975    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerName="istio-proxy"
Nov 02 10:00:15.499005 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.498982    4614 cpu_manager.go:414] "containerMap: removing container" podUID="adde7f6c-619c-4d32-9d5e-0540c56c96fc" containerName="istio-proxy"
Nov 02 10:00:15.499005 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.498987    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adde7f6c-619c-4d32-9d5e-0540c56c96fc" containerName="istio-proxy"
Nov 02 10:00:15.499005 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.498993    4614 cpu_manager.go:414] "containerMap: removing container" podUID="6f3a1400-7110-4d96-884a-f3874bf496ce" containerName="istio-proxy"
Nov 02 10:00:15.499005 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.498998    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f3a1400-7110-4d96-884a-f3874bf496ce" containerName="istio-proxy"
Nov 02 10:00:15.499235 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.499013    4614 cpu_manager.go:414] "containerMap: removing container" podUID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerName="istio-proxy"
Nov 02 10:00:15.499235 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.499018    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerName="istio-proxy"
Nov 02 10:00:15.499235 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.499094    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerName="istio-proxy"
Nov 02 10:00:15.499235 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.499100    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="adde7f6c-619c-4d32-9d5e-0540c56c96fc" containerName="istio-proxy"
Nov 02 10:00:15.499235 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.499107    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f3a1400-7110-4d96-884a-f3874bf496ce" containerName="istio-proxy"
Nov 02 10:00:15.499235 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.499118    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="125f651d-4e3c-4f0e-8748-50e55a48407d" containerName="istio-proxy"
Nov 02 10:00:15.502395 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.502375    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:15.510722 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.510698    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bd224a8ca00c\""
Nov 02 10:00:15.512469 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.512436    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 10:00:15.512469 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.512444    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 10:00:15.512650 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.512572    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 10:00:15.512650 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.512635    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-8xg25\""
Nov 02 10:00:15.513452 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.512705    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 10:00:15.515239 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.515222    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"]
Nov 02 10:00:15.587053 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.587013    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqqpx\" (UniqueName: \"kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-kube-api-access-qqqpx\") pod \"payload-processing-external-model-6cdf744db-49v9c\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:15.587229 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.587077    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume\") pod \"payload-processing-external-model-6cdf744db-49v9c\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:15.587272 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.587239    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials\") pod \"payload-processing-external-model-6cdf744db-49v9c\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:15.587309 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.587277    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-routing-overlay-volume\") pod \"payload-processing-external-model-6cdf744db-49v9c\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:15.690443 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.688044    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials\") pod \"payload-processing-external-model-6cdf744db-49v9c\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:15.690443 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.688115    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-routing-overlay-volume\") pod \"payload-processing-external-model-6cdf744db-49v9c\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:15.690443 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.688180    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqpx\" (UniqueName: \"kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-kube-api-access-qqqpx\") pod \"payload-processing-external-model-6cdf744db-49v9c\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:15.690443 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.688241    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume\") pod \"payload-processing-external-model-6cdf744db-49v9c\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:15.690443 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.689530    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume\") pod \"payload-processing-external-model-6cdf744db-49v9c\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:15.690833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.690719    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-routing-overlay-volume\") pod \"payload-processing-external-model-6cdf744db-49v9c\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:15.692316 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.692294    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials\") pod \"payload-processing-external-model-6cdf744db-49v9c\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:15.704521 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.704457    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqpx\" (UniqueName: \"kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-kube-api-access-qqqpx\") pod \"payload-processing-external-model-6cdf744db-49v9c\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") " pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:15.816616 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.816578    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:15.855471 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.855442    4614 generic.go:356] "Generic (PLEG): container finished" podID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" containerID="c9977210395dc890871595f8bac2f26b8da6ce46cdf64ad4627789f40ed0f3de" exitCode=60
Nov 02 10:00:15.855602 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.855508    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx" event={"ID":"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e","Type":"ContainerDied","Data":"c9977210395dc890871595f8bac2f26b8da6ce46cdf64ad4627789f40ed0f3de"}
Nov 02 10:00:15.855602 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.855584    4614 scope.go:122] "RemoveContainer" containerID="e2417d949b5144a9c83ed612e2a6ca0417c56392c2c8d1e7c6b21192825713c9"
Nov 02 10:00:15.855974 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:15.855953    4614 scope.go:122] "RemoveContainer" containerID="c9977210395dc890871595f8bac2f26b8da6ce46cdf64ad4627789f40ed0f3de"
Nov 02 10:00:15.856270 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:15.856242    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx_odh-ai-gateway-infra(c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx" podUID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e"
Nov 02 10:00:16.107807 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:16.107782    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"]
Nov 02 10:00:16.109900 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:00:16.109869    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606c11e8_471e_4ded_8296_be218c366d8f.slice/crio-da05106c75529ec15a38d3b39eb789a64c0a126362cafcaac5c98335e4088950 WatchSource:0}: Error finding container da05106c75529ec15a38d3b39eb789a64c0a126362cafcaac5c98335e4088950: Status 404 returned error can't find the container with id da05106c75529ec15a38d3b39eb789a64c0a126362cafcaac5c98335e4088950
Nov 02 10:00:16.576011 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:16.575970    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"]
Nov 02 10:00:16.870501 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:16.870410    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c" event={"ID":"606c11e8-471e-4ded-8296-be218c366d8f","Type":"ContainerStarted","Data":"32a34d5c0920bd503b38a639bce452ddce63b284c91b98c56f868b3236cd4b14"}
Nov 02 10:00:16.870718 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:16.870691    4614 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c" secret="" err="secret \"payload-processing-external-model-dockercfg-8xg25\" not found"
Nov 02 10:00:16.870818 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:16.870708    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c" event={"ID":"606c11e8-471e-4ded-8296-be218c366d8f","Type":"ContainerStarted","Data":"da05106c75529ec15a38d3b39eb789a64c0a126362cafcaac5c98335e4088950"}
Nov 02 10:00:16.870933 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:16.870923    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:16.888975 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:16.888937    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"]
Nov 02 10:00:16.897391 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:16.897321    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c" podStartSLOduration=1.897306714 podStartE2EDuration="1.897306714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:00:16.896156836 +0000 UTC m=+1497.728814915" watchObservedRunningTime="2026-11-02 10:00:16.897306714 +0000 UTC m=+1497.729964794"
Nov 02 10:00:16.921057 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:16.921023    4614 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bd224a8ca00c: secret "agc-cred-bd224a8ca00c" not found
Nov 02 10:00:16.921057 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:16.921048    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cdf744db-49v9c: secret "agc-cred-bd224a8ca00c" not found
Nov 02 10:00:16.921284 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:16.921105    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials podName:606c11e8-471e-4ded-8296-be218c366d8f nodeName:}" failed. No retries permitted until 2026-11-02 10:00:17.421084006 +0000 UTC m=+1498.253742078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials") pod "payload-processing-external-model-6cdf744db-49v9c" (UID: "606c11e8-471e-4ded-8296-be218c366d8f") : secret "agc-cred-bd224a8ca00c" not found
Nov 02 10:00:16.921604 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:16.920478    4614 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 02 10:00:16.921705 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:16.921640    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume podName:606c11e8-471e-4ded-8296-be218c366d8f nodeName:}" failed. No retries permitted until 2026-11-02 10:00:17.42162309 +0000 UTC m=+1498.254281150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume") pod "payload-processing-external-model-6cdf744db-49v9c" (UID: "606c11e8-471e-4ded-8296-be218c366d8f") : configmap "payload-processing-external-model-plugins" not found
Nov 02 10:00:17.050439 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:17.050415    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"
Nov 02 10:00:17.123740 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:17.123664    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e-kube-api-access-hn95t\" (UniqueName: \"kubernetes.io/projected/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e-kube-api-access-hn95t\") pod \"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e\" (UID: \"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e\") "
Nov 02 10:00:17.123887 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:17.123744    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e-maas-api-service-ca\") pod \"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e\" (UID: \"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e\") "
Nov 02 10:00:17.124151 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:17.124131    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e-maas-api-service-ca" pod "c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" (UID: "c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:00:17.125777 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:17.125748    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e-kube-api-access-hn95t" pod "c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" (UID: "c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e"). InnerVolumeSpecName "kube-api-access-hn95t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:17.225127 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:17.225089    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn95t\" (UniqueName: \"kubernetes.io/projected/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e-kube-api-access-hn95t\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:17.225127 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:17.225119    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e-maas-api-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:17.427518 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:17.427411    4614 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bd224a8ca00c: secret "agc-cred-bd224a8ca00c" not found
Nov 02 10:00:17.427518 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:17.427445    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cdf744db-49v9c: secret "agc-cred-bd224a8ca00c" not found
Nov 02 10:00:17.427518 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:17.427506    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials podName:606c11e8-471e-4ded-8296-be218c366d8f nodeName:}" failed. No retries permitted until 2026-11-02 10:00:18.427488246 +0000 UTC m=+1499.260146328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials") pod "payload-processing-external-model-6cdf744db-49v9c" (UID: "606c11e8-471e-4ded-8296-be218c366d8f") : secret "agc-cred-bd224a8ca00c" not found
Nov 02 10:00:17.427518 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:17.427422    4614 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 02 10:00:17.427819 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:17.427577    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume podName:606c11e8-471e-4ded-8296-be218c366d8f nodeName:}" failed. No retries permitted until 2026-11-02 10:00:18.427557265 +0000 UTC m=+1499.260215331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume") pod "payload-processing-external-model-6cdf744db-49v9c" (UID: "606c11e8-471e-4ded-8296-be218c366d8f") : configmap "payload-processing-external-model-plugins" not found
Nov 02 10:00:17.880729 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:17.880703    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"
Nov 02 10:00:17.881198 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:17.880740    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx" event={"ID":"c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e","Type":"ContainerDied","Data":"7101594b062955540b6bd2ce0d21807e207cc05fcf8a49226369753e785b8902"}
Nov 02 10:00:17.881198 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:17.880797    4614 scope.go:122] "RemoveContainer" containerID="c9977210395dc890871595f8bac2f26b8da6ce46cdf64ad4627789f40ed0f3de"
Nov 02 10:00:17.923467 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:17.923436    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"]
Nov 02 10:00:17.928492 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:17.928469    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-204fba-29893560-2hxvx"]
Nov 02 10:00:18.309004 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:18.308973    4614 scope.go:122] "RemoveContainer" containerID="217755abe99097f0533462f3453b26b638e29eb90db44c7baaf84570f9bebb56"
Nov 02 10:00:18.443467 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:18.443437    4614 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 02 10:00:18.443631 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:18.443518    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume podName:606c11e8-471e-4ded-8296-be218c366d8f nodeName:}" failed. No retries permitted until 2026-11-02 10:00:20.443497695 +0000 UTC m=+1501.276155776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume") pod "payload-processing-external-model-6cdf744db-49v9c" (UID: "606c11e8-471e-4ded-8296-be218c366d8f") : configmap "payload-processing-external-model-plugins" not found
Nov 02 10:00:18.444765 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:18.444659    4614 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bd224a8ca00c: secret "agc-cred-bd224a8ca00c" not found
Nov 02 10:00:18.444765 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:18.444683    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cdf744db-49v9c: secret "agc-cred-bd224a8ca00c" not found
Nov 02 10:00:18.444765 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:18.444736    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials podName:606c11e8-471e-4ded-8296-be218c366d8f nodeName:}" failed. No retries permitted until 2026-11-02 10:00:20.444718701 +0000 UTC m=+1501.277376763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials") pod "payload-processing-external-model-6cdf744db-49v9c" (UID: "606c11e8-471e-4ded-8296-be218c366d8f") : secret "agc-cred-bd224a8ca00c" not found
Nov 02 10:00:18.892626 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:18.892583    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf" event={"ID":"332aba4a-8987-44fc-bf4c-df361f601a20","Type":"ContainerStarted","Data":"d10067c0761810ea7c49e106a0156e88bdeae7db1679de038d0b39c0f5c17523"}
Nov 02 10:00:19.315980 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:19.315945    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" path="/var/lib/kubelet/pods/c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e/volumes"
Nov 02 10:00:19.758817 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:19.758747    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:00:19.778936 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:19.778907    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:00:19.900455 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:19.900423    4614 generic.go:356] "Generic (PLEG): container finished" podID="332aba4a-8987-44fc-bf4c-df361f601a20" containerID="d10067c0761810ea7c49e106a0156e88bdeae7db1679de038d0b39c0f5c17523" exitCode=60
Nov 02 10:00:19.900852 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:19.900491    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf" event={"ID":"332aba4a-8987-44fc-bf4c-df361f601a20","Type":"ContainerDied","Data":"d10067c0761810ea7c49e106a0156e88bdeae7db1679de038d0b39c0f5c17523"}
Nov 02 10:00:19.900852 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:19.900572    4614 scope.go:122] "RemoveContainer" containerID="217755abe99097f0533462f3453b26b638e29eb90db44c7baaf84570f9bebb56"
Nov 02 10:00:19.900977 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:19.900877    4614 scope.go:122] "RemoveContainer" containerID="d10067c0761810ea7c49e106a0156e88bdeae7db1679de038d0b39c0f5c17523"
Nov 02 10:00:19.901148 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:19.901125    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf_odh-ai-gateway-infra(332aba4a-8987-44fc-bf4c-df361f601a20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf" podUID="332aba4a-8987-44fc-bf4c-df361f601a20"
Nov 02 10:00:19.920221 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:19.920197    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"]
Nov 02 10:00:20.483489 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:20.483451    4614 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Nov 02 10:00:20.483671 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:20.483525    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume podName:606c11e8-471e-4ded-8296-be218c366d8f nodeName:}" failed. No retries permitted until 2026-11-02 10:00:24.483509629 +0000 UTC m=+1505.316167687 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume") pod "payload-processing-external-model-6cdf744db-49v9c" (UID: "606c11e8-471e-4ded-8296-be218c366d8f") : configmap "payload-processing-external-model-plugins" not found
Nov 02 10:00:20.483671 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:20.483533    4614 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bd224a8ca00c: secret "agc-cred-bd224a8ca00c" not found
Nov 02 10:00:20.483671 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:20.483555    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cdf744db-49v9c: secret "agc-cred-bd224a8ca00c" not found
Nov 02 10:00:20.483671 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:20.483608    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials podName:606c11e8-471e-4ded-8296-be218c366d8f nodeName:}" failed. No retries permitted until 2026-11-02 10:00:24.483592662 +0000 UTC m=+1505.316250733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials") pod "payload-processing-external-model-6cdf744db-49v9c" (UID: "606c11e8-471e-4ded-8296-be218c366d8f") : secret "agc-cred-bd224a8ca00c" not found
Nov 02 10:00:20.895465 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:20.895389    4614 scope.go:122] "RemoveContainer" containerID="c52fab50d15f562885cd202a2e60682600f0d047142403c73d03e834202473e6"
Nov 02 10:00:20.905226 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:20.905090    4614 scope.go:122] "RemoveContainer" containerID="33a1bead79e260fd5f82ce76ab1cb378743382b05eddb3d754a2013ffd6266bb"
Nov 02 10:00:20.914127 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:20.914107    4614 scope.go:122] "RemoveContainer" containerID="ec8b0e128e54dc016778feb3574c4fb917b48a074c801d8cdacf14e3560bde7d"
Nov 02 10:00:20.923058 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:20.923035    4614 scope.go:122] "RemoveContainer" containerID="c85b3845959eb3f34d9c95617c348278358e337c6f0a6d2f8ec90a73a59ac144"
Nov 02 10:00:21.074111 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:21.074082    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"
Nov 02 10:00:21.190037 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:21.189956    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/332aba4a-8987-44fc-bf4c-df361f601a20-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332aba4a-8987-44fc-bf4c-df361f601a20-maas-api-service-ca\") pod \"332aba4a-8987-44fc-bf4c-df361f601a20\" (UID: \"332aba4a-8987-44fc-bf4c-df361f601a20\") "
Nov 02 10:00:21.190193 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:21.190094    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332aba4a-8987-44fc-bf4c-df361f601a20-kube-api-access-hjx9w\" (UniqueName: \"kubernetes.io/projected/332aba4a-8987-44fc-bf4c-df361f601a20-kube-api-access-hjx9w\") pod \"332aba4a-8987-44fc-bf4c-df361f601a20\" (UID: \"332aba4a-8987-44fc-bf4c-df361f601a20\") "
Nov 02 10:00:21.190368 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:21.190329    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/332aba4a-8987-44fc-bf4c-df361f601a20-maas-api-service-ca" pod "332aba4a-8987-44fc-bf4c-df361f601a20" (UID: "332aba4a-8987-44fc-bf4c-df361f601a20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:00:21.190444 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:21.190430    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332aba4a-8987-44fc-bf4c-df361f601a20-maas-api-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:21.192235 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:21.192208    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332aba4a-8987-44fc-bf4c-df361f601a20-kube-api-access-hjx9w" pod "332aba4a-8987-44fc-bf4c-df361f601a20" (UID: "332aba4a-8987-44fc-bf4c-df361f601a20"). InnerVolumeSpecName "kube-api-access-hjx9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:21.291565 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:21.291531    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjx9w\" (UniqueName: \"kubernetes.io/projected/332aba4a-8987-44fc-bf4c-df361f601a20-kube-api-access-hjx9w\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:21.914904 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:21.914877    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"
Nov 02 10:00:21.915414 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:21.914921    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf" event={"ID":"332aba4a-8987-44fc-bf4c-df361f601a20","Type":"ContainerDied","Data":"f6c99c4f26c061d8518ab942234f2b29d738e10b09814184134860a328d23462"}
Nov 02 10:00:21.915414 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:21.914995    4614 scope.go:122] "RemoveContainer" containerID="d10067c0761810ea7c49e106a0156e88bdeae7db1679de038d0b39c0f5c17523"
Nov 02 10:00:21.933407 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:21.933377    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"]
Nov 02 10:00:21.945490 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:21.945455    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a110ef-29893560-nprhf"]
Nov 02 10:00:22.884218 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:22.884154    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c" podUID="606c11e8-471e-4ded-8296-be218c366d8f" containerName="payload-processing" containerID="cri-o://32a34d5c0920bd503b38a639bce452ddce63b284c91b98c56f868b3236cd4b14" gracePeriod=25
Nov 02 10:00:23.156339 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.156315    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:23.214518 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.214491    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-kube-api-access-qqqpx\" (UniqueName: \"kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-kube-api-access-qqqpx\") pod \"606c11e8-471e-4ded-8296-be218c366d8f\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") "
Nov 02 10:00:23.214656 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.214584    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-routing-overlay-volume\") pod \"606c11e8-471e-4ded-8296-be218c366d8f\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") "
Nov 02 10:00:23.214656 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.214630    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume\") pod \"606c11e8-471e-4ded-8296-be218c366d8f\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") "
Nov 02 10:00:23.214729 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.214658    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials\") pod \"606c11e8-471e-4ded-8296-be218c366d8f\" (UID: \"606c11e8-471e-4ded-8296-be218c366d8f\") "
Nov 02 10:00:23.214962 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.214939    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-routing-overlay-volume" pod "606c11e8-471e-4ded-8296-be218c366d8f" (UID: "606c11e8-471e-4ded-8296-be218c366d8f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:00:23.215031 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.214964    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume" pod "606c11e8-471e-4ded-8296-be218c366d8f" (UID: "606c11e8-471e-4ded-8296-be218c366d8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:00:23.215073 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.215040    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:23.215073 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.215053    4614 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/606c11e8-471e-4ded-8296-be218c366d8f-routing-overlay-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:23.216849 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.216832    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials" pod "606c11e8-471e-4ded-8296-be218c366d8f" (UID: "606c11e8-471e-4ded-8296-be218c366d8f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:23.216912 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.216844    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-kube-api-access-qqqpx" pod "606c11e8-471e-4ded-8296-be218c366d8f" (UID: "606c11e8-471e-4ded-8296-be218c366d8f"). InnerVolumeSpecName "kube-api-access-qqqpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:00:23.317252 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.317203    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="332aba4a-8987-44fc-bf4c-df361f601a20" path="/var/lib/kubelet/pods/332aba4a-8987-44fc-bf4c-df361f601a20/volumes"
Nov 02 10:00:23.324204 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.324175    4614 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-provider-credentials\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:23.324204 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.324201    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqqpx\" (UniqueName: \"kubernetes.io/projected/606c11e8-471e-4ded-8296-be218c366d8f-kube-api-access-qqqpx\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:00:23.931202 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.931123    4614 generic.go:356] "Generic (PLEG): container finished" podID="606c11e8-471e-4ded-8296-be218c366d8f" containerID="32a34d5c0920bd503b38a639bce452ddce63b284c91b98c56f868b3236cd4b14" exitCode=0
Nov 02 10:00:23.931415 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.931246    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c" event={"ID":"606c11e8-471e-4ded-8296-be218c366d8f","Type":"ContainerDied","Data":"32a34d5c0920bd503b38a639bce452ddce63b284c91b98c56f868b3236cd4b14"}
Nov 02 10:00:23.931415 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.931299    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c" event={"ID":"606c11e8-471e-4ded-8296-be218c366d8f","Type":"ContainerDied","Data":"da05106c75529ec15a38d3b39eb789a64c0a126362cafcaac5c98335e4088950"}
Nov 02 10:00:23.931415 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.931328    4614 scope.go:122] "RemoveContainer" containerID="32a34d5c0920bd503b38a639bce452ddce63b284c91b98c56f868b3236cd4b14"
Nov 02 10:00:23.931619 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.931534    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"
Nov 02 10:00:23.948580 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.948555    4614 scope.go:122] "RemoveContainer" containerID="32a34d5c0920bd503b38a639bce452ddce63b284c91b98c56f868b3236cd4b14"
Nov 02 10:00:23.949069 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:23.949033    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a34d5c0920bd503b38a639bce452ddce63b284c91b98c56f868b3236cd4b14\": container with ID starting with 32a34d5c0920bd503b38a639bce452ddce63b284c91b98c56f868b3236cd4b14 not found: ID does not exist" containerID="32a34d5c0920bd503b38a639bce452ddce63b284c91b98c56f868b3236cd4b14"
Nov 02 10:00:23.949188 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.949078    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a34d5c0920bd503b38a639bce452ddce63b284c91b98c56f868b3236cd4b14"} err="failed to get container status \"32a34d5c0920bd503b38a639bce452ddce63b284c91b98c56f868b3236cd4b14\": rpc error: code = NotFound desc = could not find container \"32a34d5c0920bd503b38a639bce452ddce63b284c91b98c56f868b3236cd4b14\": container with ID starting with 32a34d5c0920bd503b38a639bce452ddce63b284c91b98c56f868b3236cd4b14 not found: ID does not exist"
Nov 02 10:00:23.954641 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.954617    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"]
Nov 02 10:00:23.960962 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:23.960939    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdf744db-49v9c"]
Nov 02 10:00:25.316430 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:00:25.316335    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606c11e8-471e-4ded-8296-be218c366d8f" path="/var/lib/kubelet/pods/606c11e8-471e-4ded-8296-be218c366d8f/volumes"
Nov 02 10:00:39.636296 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:00:39.636254    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:01:06.690423 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690385    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"]
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690786    4614 cpu_manager.go:414] "containerMap: removing container" podUID="332aba4a-8987-44fc-bf4c-df361f601a20" containerName="cleanup"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690796    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332aba4a-8987-44fc-bf4c-df361f601a20" containerName="cleanup"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690807    4614 cpu_manager.go:414] "containerMap: removing container" podUID="332aba4a-8987-44fc-bf4c-df361f601a20" containerName="cleanup"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690812    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332aba4a-8987-44fc-bf4c-df361f601a20" containerName="cleanup"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690820    4614 cpu_manager.go:414] "containerMap: removing container" podUID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" containerName="cleanup"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690825    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" containerName="cleanup"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690847    4614 cpu_manager.go:414] "containerMap: removing container" podUID="606c11e8-471e-4ded-8296-be218c366d8f" containerName="payload-processing"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690851    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606c11e8-471e-4ded-8296-be218c366d8f" containerName="payload-processing"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690863    4614 cpu_manager.go:414] "containerMap: removing container" podUID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" containerName="cleanup"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690868    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" containerName="cleanup"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690928    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" containerName="cleanup"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690937    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a1b0ce-e12b-4227-90be-f17a736ab20d" containerName="istio-proxy"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690945    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="332aba4a-8987-44fc-bf4c-df361f601a20" containerName="cleanup"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690954    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="606c11e8-471e-4ded-8296-be218c366d8f" containerName="payload-processing"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690961    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" containerName="cleanup"
Nov 02 10:01:06.692801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.690968    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="332aba4a-8987-44fc-bf4c-df361f601a20" containerName="cleanup"
Nov 02 10:01:06.693887 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.693871    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"
Nov 02 10:01:06.701429 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.701386    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:01:06.702162 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.702136    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 10:01:06.703994 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.703972    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"]
Nov 02 10:01:06.793703 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.793664    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493137e8-645a-4e4e-a995-9a6f29d8d645-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e904c88-c6vbt\" (UID: \"493137e8-645a-4e4e-a995-9a6f29d8d645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"
Nov 02 10:01:06.793894 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.793751    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6vjs\" (UniqueName: \"kubernetes.io/projected/493137e8-645a-4e4e-a995-9a6f29d8d645-kube-api-access-z6vjs\") pod \"maas-api-revoke-sub-4e904c88-c6vbt\" (UID: \"493137e8-645a-4e4e-a995-9a6f29d8d645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"
Nov 02 10:01:06.793952 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.793886    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493137e8-645a-4e4e-a995-9a6f29d8d645-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e904c88-c6vbt\" (UID: \"493137e8-645a-4e4e-a995-9a6f29d8d645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"
Nov 02 10:01:06.895342 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.895300    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493137e8-645a-4e4e-a995-9a6f29d8d645-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e904c88-c6vbt\" (UID: \"493137e8-645a-4e4e-a995-9a6f29d8d645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"
Nov 02 10:01:06.895561 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.895392    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493137e8-645a-4e4e-a995-9a6f29d8d645-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e904c88-c6vbt\" (UID: \"493137e8-645a-4e4e-a995-9a6f29d8d645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"
Nov 02 10:01:06.895561 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.895456    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vjs\" (UniqueName: \"kubernetes.io/projected/493137e8-645a-4e4e-a995-9a6f29d8d645-kube-api-access-z6vjs\") pod \"maas-api-revoke-sub-4e904c88-c6vbt\" (UID: \"493137e8-645a-4e4e-a995-9a6f29d8d645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"
Nov 02 10:01:06.896026 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.895998    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493137e8-645a-4e4e-a995-9a6f29d8d645-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e904c88-c6vbt\" (UID: \"493137e8-645a-4e4e-a995-9a6f29d8d645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"
Nov 02 10:01:06.897780 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.897761    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493137e8-645a-4e4e-a995-9a6f29d8d645-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e904c88-c6vbt\" (UID: \"493137e8-645a-4e4e-a995-9a6f29d8d645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"
Nov 02 10:01:06.909404 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:06.909371    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6vjs\" (UniqueName: \"kubernetes.io/projected/493137e8-645a-4e4e-a995-9a6f29d8d645-kube-api-access-z6vjs\") pod \"maas-api-revoke-sub-4e904c88-c6vbt\" (UID: \"493137e8-645a-4e4e-a995-9a6f29d8d645\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"
Nov 02 10:01:07.009102 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:07.009073    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"
Nov 02 10:01:07.290527 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:07.290501    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"]
Nov 02 10:01:07.291996 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:01:07.291960    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493137e8_645a_4e4e_a995_9a6f29d8d645.slice/crio-4af812948adaa37104afba8cac7558fbfecd7be1d2a308d1b81fc3efe4fb4b95 WatchSource:0}: Error finding container 4af812948adaa37104afba8cac7558fbfecd7be1d2a308d1b81fc3efe4fb4b95: Status 404 returned error can't find the container with id 4af812948adaa37104afba8cac7558fbfecd7be1d2a308d1b81fc3efe4fb4b95
Nov 02 10:01:08.196149 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:08.196109    4614 generic.go:356] "Generic (PLEG): container finished" podID="493137e8-645a-4e4e-a995-9a6f29d8d645" containerID="0f529e052082ca789e79880f46ec132e83119defa18d1a4c70fe5386842e990c" exitCode=0
Nov 02 10:01:08.196559 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:08.196186    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt" event={"ID":"493137e8-645a-4e4e-a995-9a6f29d8d645","Type":"ContainerDied","Data":"0f529e052082ca789e79880f46ec132e83119defa18d1a4c70fe5386842e990c"}
Nov 02 10:01:08.196559 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:08.196241    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt" event={"ID":"493137e8-645a-4e4e-a995-9a6f29d8d645","Type":"ContainerStarted","Data":"4af812948adaa37104afba8cac7558fbfecd7be1d2a308d1b81fc3efe4fb4b95"}
Nov 02 10:01:08.285639 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:08.285600    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"]
Nov 02 10:01:08.296541 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:08.296514    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"]
Nov 02 10:01:08.301780 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:08.301753    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"]
Nov 02 10:01:09.354453 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:09.354428    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"
Nov 02 10:01:09.446282 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:09.446247    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/493137e8-645a-4e4e-a995-9a6f29d8d645-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493137e8-645a-4e4e-a995-9a6f29d8d645-maas-api-openshift-ca\") pod \"493137e8-645a-4e4e-a995-9a6f29d8d645\" (UID: \"493137e8-645a-4e4e-a995-9a6f29d8d645\") "
Nov 02 10:01:09.446510 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:09.446317    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493137e8-645a-4e4e-a995-9a6f29d8d645-kube-api-access-z6vjs\" (UniqueName: \"kubernetes.io/projected/493137e8-645a-4e4e-a995-9a6f29d8d645-kube-api-access-z6vjs\") pod \"493137e8-645a-4e4e-a995-9a6f29d8d645\" (UID: \"493137e8-645a-4e4e-a995-9a6f29d8d645\") "
Nov 02 10:01:09.446510 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:09.446401    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/493137e8-645a-4e4e-a995-9a6f29d8d645-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493137e8-645a-4e4e-a995-9a6f29d8d645-maas-api-xks-ca\") pod \"493137e8-645a-4e4e-a995-9a6f29d8d645\" (UID: \"493137e8-645a-4e4e-a995-9a6f29d8d645\") "
Nov 02 10:01:09.446736 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:09.446708    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493137e8-645a-4e4e-a995-9a6f29d8d645-maas-api-openshift-ca" pod "493137e8-645a-4e4e-a995-9a6f29d8d645" (UID: "493137e8-645a-4e4e-a995-9a6f29d8d645"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:01:09.448666 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:09.448633    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493137e8-645a-4e4e-a995-9a6f29d8d645-kube-api-access-z6vjs" pod "493137e8-645a-4e4e-a995-9a6f29d8d645" (UID: "493137e8-645a-4e4e-a995-9a6f29d8d645"). InnerVolumeSpecName "kube-api-access-z6vjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:09.448809 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:09.448756    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493137e8-645a-4e4e-a995-9a6f29d8d645-maas-api-xks-ca" pod "493137e8-645a-4e4e-a995-9a6f29d8d645" (UID: "493137e8-645a-4e4e-a995-9a6f29d8d645"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:01:09.548238 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:09.548209    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493137e8-645a-4e4e-a995-9a6f29d8d645-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:09.548238 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:09.548235    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6vjs\" (UniqueName: \"kubernetes.io/projected/493137e8-645a-4e4e-a995-9a6f29d8d645-kube-api-access-z6vjs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:09.548238 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:09.548246    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493137e8-645a-4e4e-a995-9a6f29d8d645-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:09.650230 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:01:09.650198    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:01:10.208366 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:10.208336    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"
Nov 02 10:01:10.208543 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:10.208345    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt" event={"ID":"493137e8-645a-4e4e-a995-9a6f29d8d645","Type":"ContainerDied","Data":"4af812948adaa37104afba8cac7558fbfecd7be1d2a308d1b81fc3efe4fb4b95"}
Nov 02 10:01:10.208543 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:10.208408    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4af812948adaa37104afba8cac7558fbfecd7be1d2a308d1b81fc3efe4fb4b95"
Nov 02 10:01:18.467735 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.467682    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"]
Nov 02 10:01:18.468325 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.468304    4614 cpu_manager.go:414] "containerMap: removing container" podUID="332aba4a-8987-44fc-bf4c-df361f601a20" containerName="cleanup"
Nov 02 10:01:18.468434 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.468327    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332aba4a-8987-44fc-bf4c-df361f601a20" containerName="cleanup"
Nov 02 10:01:18.468434 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.468339    4614 cpu_manager.go:414] "containerMap: removing container" podUID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" containerName="cleanup"
Nov 02 10:01:18.468434 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.468347    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" containerName="cleanup"
Nov 02 10:01:18.468434 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.468391    4614 cpu_manager.go:414] "containerMap: removing container" podUID="493137e8-645a-4e4e-a995-9a6f29d8d645" containerName="revoke-keys"
Nov 02 10:01:18.468434 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.468399    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493137e8-645a-4e4e-a995-9a6f29d8d645" containerName="revoke-keys"
Nov 02 10:01:18.468683 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.468545    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="c87e3019-69d9-4cbb-9c6f-5fe5bbec2d6e" containerName="cleanup"
Nov 02 10:01:18.468683 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.468563    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="332aba4a-8987-44fc-bf4c-df361f601a20" containerName="cleanup"
Nov 02 10:01:18.468683 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.468577    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="493137e8-645a-4e4e-a995-9a6f29d8d645" containerName="revoke-keys"
Nov 02 10:01:18.472129 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.472107    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"
Nov 02 10:01:18.477291 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.477259    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:01:18.478961 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.478938    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 10:01:18.480002 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.479980    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"]
Nov 02 10:01:18.585500 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.585454    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ae2c6f-4236-4d28-8c81-94de9c192785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f937c0-pfrkt\" (UID: \"03ae2c6f-4236-4d28-8c81-94de9c192785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"
Nov 02 10:01:18.585726 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.585566    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ae2c6f-4236-4d28-8c81-94de9c192785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f937c0-pfrkt\" (UID: \"03ae2c6f-4236-4d28-8c81-94de9c192785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"
Nov 02 10:01:18.585726 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.585648    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z8mf\" (UniqueName: \"kubernetes.io/projected/03ae2c6f-4236-4d28-8c81-94de9c192785-kube-api-access-4z8mf\") pod \"maas-api-revoke-sub-83f937c0-pfrkt\" (UID: \"03ae2c6f-4236-4d28-8c81-94de9c192785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"
Nov 02 10:01:18.686619 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.686581    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ae2c6f-4236-4d28-8c81-94de9c192785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f937c0-pfrkt\" (UID: \"03ae2c6f-4236-4d28-8c81-94de9c192785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"
Nov 02 10:01:18.686804 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.686626    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ae2c6f-4236-4d28-8c81-94de9c192785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f937c0-pfrkt\" (UID: \"03ae2c6f-4236-4d28-8c81-94de9c192785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"
Nov 02 10:01:18.686804 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.686688    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z8mf\" (UniqueName: \"kubernetes.io/projected/03ae2c6f-4236-4d28-8c81-94de9c192785-kube-api-access-4z8mf\") pod \"maas-api-revoke-sub-83f937c0-pfrkt\" (UID: \"03ae2c6f-4236-4d28-8c81-94de9c192785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"
Nov 02 10:01:18.687213 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.687192    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ae2c6f-4236-4d28-8c81-94de9c192785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f937c0-pfrkt\" (UID: \"03ae2c6f-4236-4d28-8c81-94de9c192785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"
Nov 02 10:01:18.689009 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.688989    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ae2c6f-4236-4d28-8c81-94de9c192785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f937c0-pfrkt\" (UID: \"03ae2c6f-4236-4d28-8c81-94de9c192785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"
Nov 02 10:01:18.699719 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.699699    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z8mf\" (UniqueName: \"kubernetes.io/projected/03ae2c6f-4236-4d28-8c81-94de9c192785-kube-api-access-4z8mf\") pod \"maas-api-revoke-sub-83f937c0-pfrkt\" (UID: \"03ae2c6f-4236-4d28-8c81-94de9c192785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"
Nov 02 10:01:18.787343 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:18.787318    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"
Nov 02 10:01:19.065830 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:19.065806    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"]
Nov 02 10:01:19.067959 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:01:19.067932    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ae2c6f_4236_4d28_8c81_94de9c192785.slice/crio-75c9499ea83172f7b49b85da77bda8e4790be5c65671e4250a8e15b3f2bc65a8 WatchSource:0}: Error finding container 75c9499ea83172f7b49b85da77bda8e4790be5c65671e4250a8e15b3f2bc65a8: Status 404 returned error can't find the container with id 75c9499ea83172f7b49b85da77bda8e4790be5c65671e4250a8e15b3f2bc65a8
Nov 02 10:01:19.255803 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:19.255769    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt" event={"ID":"03ae2c6f-4236-4d28-8c81-94de9c192785","Type":"ContainerStarted","Data":"75c9499ea83172f7b49b85da77bda8e4790be5c65671e4250a8e15b3f2bc65a8"}
Nov 02 10:01:20.262965 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:20.262931    4614 generic.go:356] "Generic (PLEG): container finished" podID="03ae2c6f-4236-4d28-8c81-94de9c192785" containerID="af8a00109bb63a01109bd46e9203c8ec6d3426125f4075e415fc8d371b2726d0" exitCode=0
Nov 02 10:01:20.263376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:20.262997    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt" event={"ID":"03ae2c6f-4236-4d28-8c81-94de9c192785","Type":"ContainerDied","Data":"af8a00109bb63a01109bd46e9203c8ec6d3426125f4075e415fc8d371b2726d0"}
Nov 02 10:01:20.986985 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:20.986953    4614 scope.go:122] "RemoveContainer" containerID="863425e043c85bf466596861be488fe952b8ac659651de0a57a097ff078e7a8f"
Nov 02 10:01:21.414731 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:21.414700    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"
Nov 02 10:01:21.516075 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:21.516041    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03ae2c6f-4236-4d28-8c81-94de9c192785-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ae2c6f-4236-4d28-8c81-94de9c192785-maas-api-xks-ca\") pod \"03ae2c6f-4236-4d28-8c81-94de9c192785\" (UID: \"03ae2c6f-4236-4d28-8c81-94de9c192785\") "
Nov 02 10:01:21.516075 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:21.516079    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ae2c6f-4236-4d28-8c81-94de9c192785-kube-api-access-4z8mf\" (UniqueName: \"kubernetes.io/projected/03ae2c6f-4236-4d28-8c81-94de9c192785-kube-api-access-4z8mf\") pod \"03ae2c6f-4236-4d28-8c81-94de9c192785\" (UID: \"03ae2c6f-4236-4d28-8c81-94de9c192785\") "
Nov 02 10:01:21.516299 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:21.516186    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ae2c6f-4236-4d28-8c81-94de9c192785-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ae2c6f-4236-4d28-8c81-94de9c192785-maas-api-openshift-ca\") pod \"03ae2c6f-4236-4d28-8c81-94de9c192785\" (UID: \"03ae2c6f-4236-4d28-8c81-94de9c192785\") "
Nov 02 10:01:21.516609 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:21.516575    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ae2c6f-4236-4d28-8c81-94de9c192785-maas-api-openshift-ca" pod "03ae2c6f-4236-4d28-8c81-94de9c192785" (UID: "03ae2c6f-4236-4d28-8c81-94de9c192785"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:01:21.518216 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:21.518190    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ae2c6f-4236-4d28-8c81-94de9c192785-maas-api-xks-ca" pod "03ae2c6f-4236-4d28-8c81-94de9c192785" (UID: "03ae2c6f-4236-4d28-8c81-94de9c192785"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:01:21.518328 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:21.518196    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ae2c6f-4236-4d28-8c81-94de9c192785-kube-api-access-4z8mf" pod "03ae2c6f-4236-4d28-8c81-94de9c192785" (UID: "03ae2c6f-4236-4d28-8c81-94de9c192785"). InnerVolumeSpecName "kube-api-access-4z8mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:21.617367 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:21.617324    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ae2c6f-4236-4d28-8c81-94de9c192785-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:21.617518 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:21.617385    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z8mf\" (UniqueName: \"kubernetes.io/projected/03ae2c6f-4236-4d28-8c81-94de9c192785-kube-api-access-4z8mf\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:21.617518 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:21.617399    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ae2c6f-4236-4d28-8c81-94de9c192785-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:22.274748 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:22.274719    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"
Nov 02 10:01:22.274748 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:22.274733    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt" event={"ID":"03ae2c6f-4236-4d28-8c81-94de9c192785","Type":"ContainerDied","Data":"75c9499ea83172f7b49b85da77bda8e4790be5c65671e4250a8e15b3f2bc65a8"}
Nov 02 10:01:22.274953 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:22.274774    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75c9499ea83172f7b49b85da77bda8e4790be5c65671e4250a8e15b3f2bc65a8"
Nov 02 10:01:23.307042 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.306986    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main" containerID="cri-o://831c2092bd42e41bb600b2addc3e6b92355667632c30cb6aa814f2600b422bf5" gracePeriod=45
Nov 02 10:01:23.316876 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.316844    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main" containerID="cri-o://efad52f5b3aa5bc5e8f8a3d1b6ad2620f7cc183d05d90f1624f6799644e6ad4e" gracePeriod=45
Nov 02 10:01:23.321887 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.321853    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main" containerID="cri-o://f4188f79adda623ff74dc4f6ee50741fc910b79000626b492dbfc1d01c890a63" gracePeriod=45
Nov 02 10:01:23.615973 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.615943    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 10:01:23.618085 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.618056    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 10:01:23.634547 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.634520    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 10:01:23.751041 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751005    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a21c87af-b381-44b0-84f8-0f7b15db29d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/a21c87af-b381-44b0-84f8-0f7b15db29d6-tls-certs\") pod \"a21c87af-b381-44b0-84f8-0f7b15db29d6\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") "
Nov 02 10:01:23.751232 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751058    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-home\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-home\") pod \"577534f3-6ae1-40fe-a27f-29084e674662\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") "
Nov 02 10:01:23.751232 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751100    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-model-cache\") pod \"e0140d2d-0778-462c-af17-bc634966b6be\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") "
Nov 02 10:01:23.751232 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751152    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-dshm\") pod \"a21c87af-b381-44b0-84f8-0f7b15db29d6\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") "
Nov 02 10:01:23.751409 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751237    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/577534f3-6ae1-40fe-a27f-29084e674662-tls-certs\" (UniqueName: \"kubernetes.io/secret/577534f3-6ae1-40fe-a27f-29084e674662-tls-certs\") pod \"577534f3-6ae1-40fe-a27f-29084e674662\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") "
Nov 02 10:01:23.751409 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751267    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-dshm\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-dshm\") pod \"577534f3-6ae1-40fe-a27f-29084e674662\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") "
Nov 02 10:01:23.751409 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751293    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0140d2d-0778-462c-af17-bc634966b6be-kube-api-access-zt9dm\" (UniqueName: \"kubernetes.io/projected/e0140d2d-0778-462c-af17-bc634966b6be-kube-api-access-zt9dm\") pod \"e0140d2d-0778-462c-af17-bc634966b6be\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") "
Nov 02 10:01:23.751409 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751342    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-tmp-dir\") pod \"e0140d2d-0778-462c-af17-bc634966b6be\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") "
Nov 02 10:01:23.751409 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751399    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577534f3-6ae1-40fe-a27f-29084e674662-kube-api-access-jrrvf\" (UniqueName: \"kubernetes.io/projected/577534f3-6ae1-40fe-a27f-29084e674662-kube-api-access-jrrvf\") pod \"577534f3-6ae1-40fe-a27f-29084e674662\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") "
Nov 02 10:01:23.751674 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751433    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-model-cache\") pod \"577534f3-6ae1-40fe-a27f-29084e674662\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") "
Nov 02 10:01:23.751674 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751443    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-home" pod "577534f3-6ae1-40fe-a27f-29084e674662" (UID: "577534f3-6ae1-40fe-a27f-29084e674662"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.751674 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751471    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-home\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-home\") pod \"e0140d2d-0778-462c-af17-bc634966b6be\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") "
Nov 02 10:01:23.751674 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751502    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-model-cache\") pod \"a21c87af-b381-44b0-84f8-0f7b15db29d6\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") "
Nov 02 10:01:23.751674 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751533    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0140d2d-0778-462c-af17-bc634966b6be-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0140d2d-0778-462c-af17-bc634966b6be-tls-certs\") pod \"e0140d2d-0778-462c-af17-bc634966b6be\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") "
Nov 02 10:01:23.751674 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751583    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-home\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-home\") pod \"a21c87af-b381-44b0-84f8-0f7b15db29d6\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") "
Nov 02 10:01:23.751674 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751614    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-tmp-dir\") pod \"577534f3-6ae1-40fe-a27f-29084e674662\" (UID: \"577534f3-6ae1-40fe-a27f-29084e674662\") "
Nov 02 10:01:23.751674 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751649    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-dshm\") pod \"e0140d2d-0778-462c-af17-bc634966b6be\" (UID: \"e0140d2d-0778-462c-af17-bc634966b6be\") "
Nov 02 10:01:23.752096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751677    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21c87af-b381-44b0-84f8-0f7b15db29d6-kube-api-access-wnbq6\" (UniqueName: \"kubernetes.io/projected/a21c87af-b381-44b0-84f8-0f7b15db29d6-kube-api-access-wnbq6\") pod \"a21c87af-b381-44b0-84f8-0f7b15db29d6\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") "
Nov 02 10:01:23.752096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.751724    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-tmp-dir\") pod \"a21c87af-b381-44b0-84f8-0f7b15db29d6\" (UID: \"a21c87af-b381-44b0-84f8-0f7b15db29d6\") "
Nov 02 10:01:23.752206 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.752148    4614 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-home\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.752266 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.752209    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-model-cache" pod "a21c87af-b381-44b0-84f8-0f7b15db29d6" (UID: "a21c87af-b381-44b0-84f8-0f7b15db29d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.752486 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.752370    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-model-cache" pod "e0140d2d-0778-462c-af17-bc634966b6be" (UID: "e0140d2d-0778-462c-af17-bc634966b6be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.752486 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.752379    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-tmp-dir" pod "a21c87af-b381-44b0-84f8-0f7b15db29d6" (UID: "a21c87af-b381-44b0-84f8-0f7b15db29d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.752789 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.752766    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-tmp-dir" pod "e0140d2d-0778-462c-af17-bc634966b6be" (UID: "e0140d2d-0778-462c-af17-bc634966b6be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.753036 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.752971    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-home" pod "e0140d2d-0778-462c-af17-bc634966b6be" (UID: "e0140d2d-0778-462c-af17-bc634966b6be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.753036 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.753000    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-tmp-dir" pod "577534f3-6ae1-40fe-a27f-29084e674662" (UID: "577534f3-6ae1-40fe-a27f-29084e674662"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.753250 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.752910    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-model-cache" pod "577534f3-6ae1-40fe-a27f-29084e674662" (UID: "577534f3-6ae1-40fe-a27f-29084e674662"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.753250 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.753205    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-home" pod "a21c87af-b381-44b0-84f8-0f7b15db29d6" (UID: "a21c87af-b381-44b0-84f8-0f7b15db29d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.753985 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.753957    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-dshm" pod "577534f3-6ae1-40fe-a27f-29084e674662" (UID: "577534f3-6ae1-40fe-a27f-29084e674662"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.754176 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.754150    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/577534f3-6ae1-40fe-a27f-29084e674662-tls-certs" pod "577534f3-6ae1-40fe-a27f-29084e674662" (UID: "577534f3-6ae1-40fe-a27f-29084e674662"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:01:23.754928 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.754903    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21c87af-b381-44b0-84f8-0f7b15db29d6-tls-certs" pod "a21c87af-b381-44b0-84f8-0f7b15db29d6" (UID: "a21c87af-b381-44b0-84f8-0f7b15db29d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:01:23.755261 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.755235    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-dshm" pod "e0140d2d-0778-462c-af17-bc634966b6be" (UID: "e0140d2d-0778-462c-af17-bc634966b6be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.755524 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.755498    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-dshm" pod "a21c87af-b381-44b0-84f8-0f7b15db29d6" (UID: "a21c87af-b381-44b0-84f8-0f7b15db29d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:23.755671 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.755649    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0140d2d-0778-462c-af17-bc634966b6be-kube-api-access-zt9dm" pod "e0140d2d-0778-462c-af17-bc634966b6be" (UID: "e0140d2d-0778-462c-af17-bc634966b6be"). InnerVolumeSpecName "kube-api-access-zt9dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:23.755856 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.755835    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21c87af-b381-44b0-84f8-0f7b15db29d6-kube-api-access-wnbq6" pod "a21c87af-b381-44b0-84f8-0f7b15db29d6" (UID: "a21c87af-b381-44b0-84f8-0f7b15db29d6"). InnerVolumeSpecName "kube-api-access-wnbq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:23.756252 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.756231    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0140d2d-0778-462c-af17-bc634966b6be-tls-certs" pod "e0140d2d-0778-462c-af17-bc634966b6be" (UID: "e0140d2d-0778-462c-af17-bc634966b6be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:01:23.756760 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.756735    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577534f3-6ae1-40fe-a27f-29084e674662-kube-api-access-jrrvf" pod "577534f3-6ae1-40fe-a27f-29084e674662" (UID: "577534f3-6ae1-40fe-a27f-29084e674662"). InnerVolumeSpecName "kube-api-access-jrrvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:23.832660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.831293    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"]
Nov 02 10:01:23.832660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.832102    4614 cpu_manager.go:414] "containerMap: removing container" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main"
Nov 02 10:01:23.832660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.832124    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main"
Nov 02 10:01:23.832660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.832181    4614 cpu_manager.go:414] "containerMap: removing container" podUID="03ae2c6f-4236-4d28-8c81-94de9c192785" containerName="revoke-keys"
Nov 02 10:01:23.832660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.832191    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ae2c6f-4236-4d28-8c81-94de9c192785" containerName="revoke-keys"
Nov 02 10:01:23.832660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.832207    4614 cpu_manager.go:414] "containerMap: removing container" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main"
Nov 02 10:01:23.832660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.832214    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main"
Nov 02 10:01:23.832660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.832224    4614 cpu_manager.go:414] "containerMap: removing container" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main"
Nov 02 10:01:23.832660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.832248    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main"
Nov 02 10:01:23.832660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.832404    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerName="main"
Nov 02 10:01:23.832660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.832422    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ae2c6f-4236-4d28-8c81-94de9c192785" containerName="revoke-keys"
Nov 02 10:01:23.832660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.832451    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0140d2d-0778-462c-af17-bc634966b6be" containerName="main"
Nov 02 10:01:23.832660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.832460    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="577534f3-6ae1-40fe-a27f-29084e674662" containerName="main"
Nov 02 10:01:23.837751 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.837731    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:23.853728 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.853698    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"]
Nov 02 10:01:23.854435 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854414    4614 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21c87af-b381-44b0-84f8-0f7b15db29d6-tls-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854538 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854439    4614 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-model-cache\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854538 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854454    4614 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-dshm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854538 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854469    4614 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/577534f3-6ae1-40fe-a27f-29084e674662-tls-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854538 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854481    4614 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-dshm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854538 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854495    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt9dm\" (UniqueName: \"kubernetes.io/projected/e0140d2d-0778-462c-af17-bc634966b6be-kube-api-access-zt9dm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854538 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854508    4614 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-tmp-dir\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854538 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854522    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrrvf\" (UniqueName: \"kubernetes.io/projected/577534f3-6ae1-40fe-a27f-29084e674662-kube-api-access-jrrvf\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854538 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854535    4614 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-model-cache\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854951 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854548    4614 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-home\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854951 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854561    4614 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-model-cache\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854951 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854573    4614 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0140d2d-0778-462c-af17-bc634966b6be-tls-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854951 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854586    4614 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-home\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854951 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854600    4614 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/577534f3-6ae1-40fe-a27f-29084e674662-tmp-dir\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854951 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854612    4614 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0140d2d-0778-462c-af17-bc634966b6be-dshm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854951 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854625    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnbq6\" (UniqueName: \"kubernetes.io/projected/a21c87af-b381-44b0-84f8-0f7b15db29d6-kube-api-access-wnbq6\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.854951 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.854638    4614 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21c87af-b381-44b0-84f8-0f7b15db29d6-tmp-dir\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:23.959376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.957879    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxwtc\" (UniqueName: \"kubernetes.io/projected/06f71c73-0d76-4483-b204-595f241c0a89-kube-api-access-hxwtc\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:23.959376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.957946    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:23.959376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.957988    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:23.959376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.958014    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f71c73-0d76-4483-b204-595f241c0a89-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:23.959376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.958054    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f71c73-0d76-4483-b204-595f241c0a89-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:23.959376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.958100    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:23.959376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.958132    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:23.959376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.958158    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f71c73-0d76-4483-b204-595f241c0a89-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:23.959376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:23.958187    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.058961 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.058923    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.058961 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.058964    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f71c73-0d76-4483-b204-595f241c0a89-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.059225 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.058993    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f71c73-0d76-4483-b204-595f241c0a89-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.059225 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.059031    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.059225 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.059051    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.059225 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.059066    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f71c73-0d76-4483-b204-595f241c0a89-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.059225 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.059092    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.059549 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.059446    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-workload-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.059619 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.059579    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxwtc\" (UniqueName: \"kubernetes.io/projected/06f71c73-0d76-4483-b204-595f241c0a89-kube-api-access-hxwtc\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.059679 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.059622    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.059679 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.059657    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-istio-data\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.059989 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.059958    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-credential-socket\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.060118 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.060075    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-workload-certs\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.060303 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.060280    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f71c73-0d76-4483-b204-595f241c0a89-istiod-ca-cert\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.061854 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.061833    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-istio-envoy\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.061987 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.061967    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f71c73-0d76-4483-b204-595f241c0a89-istio-podinfo\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.071622 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.071602    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f71c73-0d76-4483-b204-595f241c0a89-istio-token\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.072426 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.072409    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxwtc\" (UniqueName: \"kubernetes.io/projected/06f71c73-0d76-4483-b204-595f241c0a89-kube-api-access-hxwtc\") pod \"e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") " pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.155450 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.155388    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:24.286608 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.286574    4614 generic.go:356] "Generic (PLEG): container finished" podID="577534f3-6ae1-40fe-a27f-29084e674662" containerID="efad52f5b3aa5bc5e8f8a3d1b6ad2620f7cc183d05d90f1624f6799644e6ad4e" exitCode=0
Nov 02 10:01:24.286784 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.286625    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"
Nov 02 10:01:24.286784 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.286659    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" event={"ID":"577534f3-6ae1-40fe-a27f-29084e674662","Type":"ContainerDied","Data":"efad52f5b3aa5bc5e8f8a3d1b6ad2620f7cc183d05d90f1624f6799644e6ad4e"}
Nov 02 10:01:24.286784 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.286712    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl" event={"ID":"577534f3-6ae1-40fe-a27f-29084e674662","Type":"ContainerDied","Data":"cfb142e0c1e6f408053f79d44df8317fe2e9ab04a3c4ecf0a54d30b974ee1163"}
Nov 02 10:01:24.286784 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.286737    4614 scope.go:122] "RemoveContainer" containerID="efad52f5b3aa5bc5e8f8a3d1b6ad2620f7cc183d05d90f1624f6799644e6ad4e"
Nov 02 10:01:24.288062 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.288042    4614 generic.go:356] "Generic (PLEG): container finished" podID="e0140d2d-0778-462c-af17-bc634966b6be" containerID="f4188f79adda623ff74dc4f6ee50741fc910b79000626b492dbfc1d01c890a63" exitCode=0
Nov 02 10:01:24.288170 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.288089    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"
Nov 02 10:01:24.288170 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.288129    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" event={"ID":"e0140d2d-0778-462c-af17-bc634966b6be","Type":"ContainerDied","Data":"f4188f79adda623ff74dc4f6ee50741fc910b79000626b492dbfc1d01c890a63"}
Nov 02 10:01:24.288170 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.288167    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9" event={"ID":"e0140d2d-0778-462c-af17-bc634966b6be","Type":"ContainerDied","Data":"7204847b56817c3a5e6c8cf99a6d0b41baed477ae62d92840e8cb2d348c7147b"}
Nov 02 10:01:24.289492 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.289470    4614 generic.go:356] "Generic (PLEG): container finished" podID="a21c87af-b381-44b0-84f8-0f7b15db29d6" containerID="831c2092bd42e41bb600b2addc3e6b92355667632c30cb6aa814f2600b422bf5" exitCode=0
Nov 02 10:01:24.289605 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.289499    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" event={"ID":"a21c87af-b381-44b0-84f8-0f7b15db29d6","Type":"ContainerDied","Data":"831c2092bd42e41bb600b2addc3e6b92355667632c30cb6aa814f2600b422bf5"}
Nov 02 10:01:24.289605 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.289537    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29" event={"ID":"a21c87af-b381-44b0-84f8-0f7b15db29d6","Type":"ContainerDied","Data":"5b3bf6a3375ef24f0cb9790eb9cff7c9a90ac7fb93f2b8c631d9c02dfa855362"}
Nov 02 10:01:24.289605 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.289512    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"
Nov 02 10:01:24.297511 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.297489    4614 scope.go:122] "RemoveContainer" containerID="efad52f5b3aa5bc5e8f8a3d1b6ad2620f7cc183d05d90f1624f6799644e6ad4e"
Nov 02 10:01:24.297803 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:01:24.297780    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efad52f5b3aa5bc5e8f8a3d1b6ad2620f7cc183d05d90f1624f6799644e6ad4e\": container with ID starting with efad52f5b3aa5bc5e8f8a3d1b6ad2620f7cc183d05d90f1624f6799644e6ad4e not found: ID does not exist" containerID="efad52f5b3aa5bc5e8f8a3d1b6ad2620f7cc183d05d90f1624f6799644e6ad4e"
Nov 02 10:01:24.297922 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.297813    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efad52f5b3aa5bc5e8f8a3d1b6ad2620f7cc183d05d90f1624f6799644e6ad4e"} err="failed to get container status \"efad52f5b3aa5bc5e8f8a3d1b6ad2620f7cc183d05d90f1624f6799644e6ad4e\": rpc error: code = NotFound desc = could not find container \"efad52f5b3aa5bc5e8f8a3d1b6ad2620f7cc183d05d90f1624f6799644e6ad4e\": container with ID starting with efad52f5b3aa5bc5e8f8a3d1b6ad2620f7cc183d05d90f1624f6799644e6ad4e not found: ID does not exist"
Nov 02 10:01:24.297922 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.297835    4614 scope.go:122] "RemoveContainer" containerID="f4188f79adda623ff74dc4f6ee50741fc910b79000626b492dbfc1d01c890a63"
Nov 02 10:01:24.307564 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.307515    4614 scope.go:122] "RemoveContainer" containerID="f4188f79adda623ff74dc4f6ee50741fc910b79000626b492dbfc1d01c890a63"
Nov 02 10:01:24.307885 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:01:24.307835    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4188f79adda623ff74dc4f6ee50741fc910b79000626b492dbfc1d01c890a63\": container with ID starting with f4188f79adda623ff74dc4f6ee50741fc910b79000626b492dbfc1d01c890a63 not found: ID does not exist" containerID="f4188f79adda623ff74dc4f6ee50741fc910b79000626b492dbfc1d01c890a63"
Nov 02 10:01:24.307885 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.307862    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4188f79adda623ff74dc4f6ee50741fc910b79000626b492dbfc1d01c890a63"} err="failed to get container status \"f4188f79adda623ff74dc4f6ee50741fc910b79000626b492dbfc1d01c890a63\": rpc error: code = NotFound desc = could not find container \"f4188f79adda623ff74dc4f6ee50741fc910b79000626b492dbfc1d01c890a63\": container with ID starting with f4188f79adda623ff74dc4f6ee50741fc910b79000626b492dbfc1d01c890a63 not found: ID does not exist"
Nov 02 10:01:24.307885 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.307882    4614 scope.go:122] "RemoveContainer" containerID="831c2092bd42e41bb600b2addc3e6b92355667632c30cb6aa814f2600b422bf5"
Nov 02 10:01:24.315560 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.315544    4614 scope.go:122] "RemoveContainer" containerID="831c2092bd42e41bb600b2addc3e6b92355667632c30cb6aa814f2600b422bf5"
Nov 02 10:01:24.315855 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:01:24.315813    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"831c2092bd42e41bb600b2addc3e6b92355667632c30cb6aa814f2600b422bf5\": container with ID starting with 831c2092bd42e41bb600b2addc3e6b92355667632c30cb6aa814f2600b422bf5 not found: ID does not exist" containerID="831c2092bd42e41bb600b2addc3e6b92355667632c30cb6aa814f2600b422bf5"
Nov 02 10:01:24.315935 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.315863    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"831c2092bd42e41bb600b2addc3e6b92355667632c30cb6aa814f2600b422bf5"} err="failed to get container status \"831c2092bd42e41bb600b2addc3e6b92355667632c30cb6aa814f2600b422bf5\": rpc error: code = NotFound desc = could not find container \"831c2092bd42e41bb600b2addc3e6b92355667632c30cb6aa814f2600b422bf5\": container with ID starting with 831c2092bd42e41bb600b2addc3e6b92355667632c30cb6aa814f2600b422bf5 not found: ID does not exist"
Nov 02 10:01:24.321894 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.321873    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"]
Nov 02 10:01:24.326556 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.326458    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/distinct-w3-a110ef-kserve-56bf8cd4b6-6zn29"]
Nov 02 10:01:24.339118 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.339099    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"]
Nov 02 10:01:24.342197 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.342176    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/premium-w3-a110ef-kserve-784d885ff9-ck4kl"]
Nov 02 10:01:24.354556 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.354534    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"]
Nov 02 10:01:24.360510 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.360489    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a110ef/simulator-w3-a110ef-kserve-5874d9cf94-bn2f9"]
Nov 02 10:01:24.434428 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.434402    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"]
Nov 02 10:01:24.436702 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:01:24.436676    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f71c73_0d76_4483_b204_595f241c0a89.slice/crio-f40ababa2b9fb22b537b3e4e0d6abf4f5f5117ed4f7f01ce9387622a594cae14 WatchSource:0}: Error finding container f40ababa2b9fb22b537b3e4e0d6abf4f5f5117ed4f7f01ce9387622a594cae14: Status 404 returned error can't find the container with id f40ababa2b9fb22b537b3e4e0d6abf4f5f5117ed4f7f01ce9387622a594cae14
Nov 02 10:01:24.439344 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.439320    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:01:24.439424 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.439403    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:01:24.439458 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:24.439436    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:01:25.314830 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:25.314800    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="577534f3-6ae1-40fe-a27f-29084e674662" path="/var/lib/kubelet/pods/577534f3-6ae1-40fe-a27f-29084e674662/volumes"
Nov 02 10:01:25.315200 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:25.315186    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a21c87af-b381-44b0-84f8-0f7b15db29d6" path="/var/lib/kubelet/pods/a21c87af-b381-44b0-84f8-0f7b15db29d6/volumes"
Nov 02 10:01:25.315581 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:25.315566    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0140d2d-0778-462c-af17-bc634966b6be" path="/var/lib/kubelet/pods/e0140d2d-0778-462c-af17-bc634966b6be/volumes"
Nov 02 10:01:25.315827 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:25.315813    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh" event={"ID":"06f71c73-0d76-4483-b204-595f241c0a89","Type":"ContainerStarted","Data":"c7fe0aa36bdc05573a1ef1e762b466a297a4aaa6081e7d0721da5f5f7480f0ed"}
Nov 02 10:01:25.315876 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:25.315850    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh" event={"ID":"06f71c73-0d76-4483-b204-595f241c0a89","Type":"ContainerStarted","Data":"f40ababa2b9fb22b537b3e4e0d6abf4f5f5117ed4f7f01ce9387622a594cae14"}
Nov 02 10:01:25.352990 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:25.352938    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh" podStartSLOduration=2.352921039 podStartE2EDuration="2.352921039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:01:25.351875891 +0000 UTC m=+1566.184533972" watchObservedRunningTime="2026-11-02 10:01:25.352921039 +0000 UTC m=+1566.185579118"
Nov 02 10:01:26.156689 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:26.156652    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:26.158332 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:26.158304    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 02 10:01:26.158492 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:26.158366    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh" podUID="06f71c73-0d76-4483-b204-595f241c0a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 02 10:01:27.156649 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:27.156610    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 02 10:01:27.157053 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:27.156663    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh" podUID="06f71c73-0d76-4483-b204-595f241c0a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 02 10:01:28.155885 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:28.155839    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 02 10:01:28.156095 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:28.155908    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh" podUID="06f71c73-0d76-4483-b204-595f241c0a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 02 10:01:29.156853 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:29.156815    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 02 10:01:29.157253 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:29.156871    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh" podUID="06f71c73-0d76-4483-b204-595f241c0a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 02 10:01:30.156737 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:30.156651    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 02 10:01:30.156737 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:30.156707    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh" podUID="06f71c73-0d76-4483-b204-595f241c0a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 02 10:01:31.155971 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:31.155931    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 02 10:01:31.156478 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:31.155983    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh" podUID="06f71c73-0d76-4483-b204-595f241c0a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 02 10:01:33.162128 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:33.161447    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:01:33.162128 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:33.161531    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh" podUID="06f71c73-0d76-4483-b204-595f241c0a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:01:34.212465 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:34.209495    4614 patch_prober.go:29] interesting pod/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:01:34.212465 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:34.209577    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh" podUID="06f71c73-0d76-4483-b204-595f241c0a89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:01:35.107233 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:35.107192    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:35.107615 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:35.107590    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:35.108407 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:35.108386    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:39.673758 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:01:39.673671    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:01:39.935958 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:39.935880    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"]
Nov 02 10:01:39.938942 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:39.938918    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:39.947107 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:39.947087    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-f77d6b5159d1\""
Nov 02 10:01:39.947373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:39.947329    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"kube-root-ca.crt\""
Nov 02 10:01:39.950196 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:39.949463    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"routing-overlay\""
Nov 02 10:01:39.950196 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:39.949500    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-plugins-e2e-extmod-b-204fba\""
Nov 02 10:01:39.950196 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:39.949557    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"openshift-service-ca.crt\""
Nov 02 10:01:39.950527 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:39.950510    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-e2e--e7669bbb-dockercfg-s8xkv\""
Nov 02 10:01:39.951923 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:39.951904    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"]
Nov 02 10:01:40.096238 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.096200    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9z5w\" (UniqueName: \"kubernetes.io/projected/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-kube-api-access-g9z5w\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:40.096456 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.096307    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:40.096456 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.096365    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:40.096589 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.096481    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:40.197899 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.197813    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:40.198076 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.197933    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9z5w\" (UniqueName: \"kubernetes.io/projected/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-kube-api-access-g9z5w\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:40.198076 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.197987    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:40.198076 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.198033    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:40.198648 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.198584    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:40.199028 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.199001    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:40.200811 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.200788    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:40.211869 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.211842    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9z5w\" (UniqueName: \"kubernetes.io/projected/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-kube-api-access-g9z5w\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:40.252857 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.252821    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:40.376112 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.376075    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"]
Nov 02 10:01:40.379859 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.379843    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:40.388276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.388249    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"openshift-service-ca.crt\""
Nov 02 10:01:40.388546 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.388294    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"routing-overlay\""
Nov 02 10:01:40.388546 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.388293    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"kube-root-ca.crt\""
Nov 02 10:01:40.388546 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.388383    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-plugins-e2e-extmod-a-629129\""
Nov 02 10:01:40.391637 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.391618    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"]
Nov 02 10:01:40.491408 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.489761    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-e2e--7be3f336-dockercfg-zbbxs\""
Nov 02 10:01:40.491408 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.489811    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-8eb9fb76a028\""
Nov 02 10:01:40.506376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.501881    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35561b22-9eca-405a-b437-e0ad2586d797-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7487czqfj\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:40.506376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.501949    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m85jn\" (UniqueName: \"kubernetes.io/projected/35561b22-9eca-405a-b437-e0ad2586d797-kube-api-access-m85jn\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7487czqfj\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:40.506376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.502219    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35561b22-9eca-405a-b437-e0ad2586d797-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7487czqfj\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:40.506376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.502267    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35561b22-9eca-405a-b437-e0ad2586d797-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7487czqfj\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:40.546565 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.546532    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"]
Nov 02 10:01:40.550750 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:01:40.550720    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbddbc2f2_63b0_4267_8bf9_b54ab2d1b874.slice/crio-728e787f402b2e5a840cc6197f7f8935dfb2498a8c97fe034173de849c5a5167 WatchSource:0}: Error finding container 728e787f402b2e5a840cc6197f7f8935dfb2498a8c97fe034173de849c5a5167: Status 404 returned error can't find the container with id 728e787f402b2e5a840cc6197f7f8935dfb2498a8c97fe034173de849c5a5167
Nov 02 10:01:40.602937 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.602910    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35561b22-9eca-405a-b437-e0ad2586d797-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7487czqfj\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:40.603067 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.602981    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85jn\" (UniqueName: \"kubernetes.io/projected/35561b22-9eca-405a-b437-e0ad2586d797-kube-api-access-m85jn\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7487czqfj\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:40.603132 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.603111    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35561b22-9eca-405a-b437-e0ad2586d797-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7487czqfj\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:40.603188 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.603154    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35561b22-9eca-405a-b437-e0ad2586d797-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7487czqfj\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:40.603834 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.603806    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35561b22-9eca-405a-b437-e0ad2586d797-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7487czqfj\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:40.603950 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.603831    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35561b22-9eca-405a-b437-e0ad2586d797-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7487czqfj\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:40.605900 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.605867    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35561b22-9eca-405a-b437-e0ad2586d797-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7487czqfj\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:40.617400 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.617381    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m85jn\" (UniqueName: \"kubernetes.io/projected/35561b22-9eca-405a-b437-e0ad2586d797-kube-api-access-m85jn\") pod \"payload-processing-external-model-e2e-extmod-a-629129-7487czqfj\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:40.694275 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.694247    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:40.981102 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:40.981071    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"]
Nov 02 10:01:40.983390 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:01:40.983361    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35561b22_9eca_405a_b437_e0ad2586d797.slice/crio-1a22b9191ba6163fe62b1a4ce4f87b72affd304a23361d1b7478ba72ab7ef05e WatchSource:0}: Error finding container 1a22b9191ba6163fe62b1a4ce4f87b72affd304a23361d1b7478ba72ab7ef05e: Status 404 returned error can't find the container with id 1a22b9191ba6163fe62b1a4ce4f87b72affd304a23361d1b7478ba72ab7ef05e
Nov 02 10:01:41.505600 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:41.505554    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj" event={"ID":"35561b22-9eca-405a-b437-e0ad2586d797","Type":"ContainerStarted","Data":"290ec5df2b522a5d7a088177c76eeb933c7fb18f8b0f67d752a88fe1d6f72aa2"}
Nov 02 10:01:41.505772 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:41.505627    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj" event={"ID":"35561b22-9eca-405a-b437-e0ad2586d797","Type":"ContainerStarted","Data":"1a22b9191ba6163fe62b1a4ce4f87b72affd304a23361d1b7478ba72ab7ef05e"}
Nov 02 10:01:41.505772 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:41.505764    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:41.507158 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:41.507137    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8" event={"ID":"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874","Type":"ContainerStarted","Data":"ed08e2e2889abe87ab235c5569ce622bd12c86950479961b7b2a16cec204dabd"}
Nov 02 10:01:41.507268 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:41.507180    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8" event={"ID":"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874","Type":"ContainerStarted","Data":"728e787f402b2e5a840cc6197f7f8935dfb2498a8c97fe034173de849c5a5167"}
Nov 02 10:01:41.507268 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:41.507223    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:41.530516 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:41.530477    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj" podStartSLOduration=1.5304654549999999 podStartE2EDuration="1.530465455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:01:41.527411474 +0000 UTC m=+1582.360069553" watchObservedRunningTime="2026-11-02 10:01:41.530465455 +0000 UTC m=+1582.363123534"
Nov 02 10:01:41.548090 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:41.548048    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8" podStartSLOduration=2.548036042 podStartE2EDuration="2.548036042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:01:41.547044504 +0000 UTC m=+1582.379702586" watchObservedRunningTime="2026-11-02 10:01:41.548036042 +0000 UTC m=+1582.380694151"
Nov 02 10:01:50.481160 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:50.481030    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"]
Nov 02 10:01:50.481808 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:50.481306    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492" podUID="ae2706aa-d41a-463a-acf6-79c6b2705f9c" containerName="istio-proxy" containerID="cri-o://48f208575364f300d286b90bb0c7622665ffb21b9a3df7fc159bbad69c181246" gracePeriod=30
Nov 02 10:01:50.486054 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:50.486019    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"]
Nov 02 10:01:50.486449 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:50.486403    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" podUID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" containerName="istio-proxy" containerID="cri-o://c27db4bf0858e319748be554a041cef4a225f4879d5dc35f01f26e6754f0bd38" gracePeriod=30
Nov 02 10:01:50.489317 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:50.489290    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"]
Nov 02 10:01:50.489702 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:50.489640    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh" podUID="06f71c73-0d76-4483-b204-595f241c0a89" containerName="istio-proxy" containerID="cri-o://c7fe0aa36bdc05573a1ef1e762b466a297a4aaa6081e7d0721da5f5f7480f0ed" gracePeriod=30
Nov 02 10:01:50.501473 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:50.501449    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"]
Nov 02 10:01:50.558520 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:50.558487    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerName="istio-proxy" containerID="cri-o://7ddb4b3487498c3ca60b67429bc4fc1b60f9427141f4022f4c4cc3701778086c" gracePeriod=30
Nov 02 10:01:52.515819 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:52.515792    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:01:52.516225 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:52.515862    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:01:55.833234 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.833207    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 10:01:55.853516 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.853488    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.896441    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-workload-socket\") pod \"06f71c73-0d76-4483-b204-595f241c0a89\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.896497    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-podinfo\") pod \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.896532    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-credential-socket\") pod \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.896596    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-envoy\") pod \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.896638    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-data\") pod \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.896676    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2706aa-d41a-463a-acf6-79c6b2705f9c-kube-api-access-m4j7z\" (UniqueName: \"kubernetes.io/projected/ae2706aa-d41a-463a-acf6-79c6b2705f9c-kube-api-access-m4j7z\") pod \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.896726    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-workload-certs\") pod \"06f71c73-0d76-4483-b204-595f241c0a89\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.896774    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istiod-ca-cert\") pod \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.896810    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-istio-data\") pod \"06f71c73-0d76-4483-b204-595f241c0a89\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.896846    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-credential-socket\") pod \"06f71c73-0d76-4483-b204-595f241c0a89\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.896886    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06f71c73-0d76-4483-b204-595f241c0a89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f71c73-0d76-4483-b204-595f241c0a89-istio-podinfo\") pod \"06f71c73-0d76-4483-b204-595f241c0a89\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.896914    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f71c73-0d76-4483-b204-595f241c0a89-kube-api-access-hxwtc\" (UniqueName: \"kubernetes.io/projected/06f71c73-0d76-4483-b204-595f241c0a89-kube-api-access-hxwtc\") pod \"06f71c73-0d76-4483-b204-595f241c0a89\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.896949    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-workload-socket\") pod \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.896975    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-workload-certs\") pod \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.897003    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f71c73-0d76-4483-b204-595f241c0a89-istio-token\" (UniqueName: \"kubernetes.io/projected/06f71c73-0d76-4483-b204-595f241c0a89-istio-token\") pod \"06f71c73-0d76-4483-b204-595f241c0a89\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.897047    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-token\" (UniqueName: \"kubernetes.io/projected/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-token\") pod \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\" (UID: \"ae2706aa-d41a-463a-acf6-79c6b2705f9c\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.897123    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f71c73-0d76-4483-b204-595f241c0a89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f71c73-0d76-4483-b204-595f241c0a89-istiod-ca-cert\") pod \"06f71c73-0d76-4483-b204-595f241c0a89\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.897176    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-istio-envoy\") pod \"06f71c73-0d76-4483-b204-595f241c0a89\" (UID: \"06f71c73-0d76-4483-b204-595f241c0a89\") "
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.899633    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-workload-socket" pod "06f71c73-0d76-4483-b204-595f241c0a89" (UID: "06f71c73-0d76-4483-b204-595f241c0a89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.901416    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istiod-ca-cert" pod "ae2706aa-d41a-463a-acf6-79c6b2705f9c" (UID: "ae2706aa-d41a-463a-acf6-79c6b2705f9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.901698    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-workload-certs" pod "06f71c73-0d76-4483-b204-595f241c0a89" (UID: "06f71c73-0d76-4483-b204-595f241c0a89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.901882    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-istio-data" pod "06f71c73-0d76-4483-b204-595f241c0a89" (UID: "06f71c73-0d76-4483-b204-595f241c0a89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.902053    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-data" pod "ae2706aa-d41a-463a-acf6-79c6b2705f9c" (UID: "ae2706aa-d41a-463a-acf6-79c6b2705f9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.902373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.902327    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-workload-socket" pod "ae2706aa-d41a-463a-acf6-79c6b2705f9c" (UID: "ae2706aa-d41a-463a-acf6-79c6b2705f9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.929135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.902563    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-credential-socket" pod "ae2706aa-d41a-463a-acf6-79c6b2705f9c" (UID: "ae2706aa-d41a-463a-acf6-79c6b2705f9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.929135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.912429    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-workload-certs" pod "ae2706aa-d41a-463a-acf6-79c6b2705f9c" (UID: "ae2706aa-d41a-463a-acf6-79c6b2705f9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.929135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.912522    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-istio-envoy" pod "06f71c73-0d76-4483-b204-595f241c0a89" (UID: "06f71c73-0d76-4483-b204-595f241c0a89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.929135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.916717    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f71c73-0d76-4483-b204-595f241c0a89-kube-api-access-hxwtc" pod "06f71c73-0d76-4483-b204-595f241c0a89" (UID: "06f71c73-0d76-4483-b204-595f241c0a89"). InnerVolumeSpecName "kube-api-access-hxwtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:55.929135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.917763    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2706aa-d41a-463a-acf6-79c6b2705f9c-kube-api-access-m4j7z" pod "ae2706aa-d41a-463a-acf6-79c6b2705f9c" (UID: "ae2706aa-d41a-463a-acf6-79c6b2705f9c"). InnerVolumeSpecName "kube-api-access-m4j7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:55.929135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.918193    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-podinfo" pod "ae2706aa-d41a-463a-acf6-79c6b2705f9c" (UID: "ae2706aa-d41a-463a-acf6-79c6b2705f9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:01:55.929135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.922963    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06f71c73-0d76-4483-b204-595f241c0a89-istio-podinfo" pod "06f71c73-0d76-4483-b204-595f241c0a89" (UID: "06f71c73-0d76-4483-b204-595f241c0a89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:01:55.929135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.923575    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-envoy" pod "ae2706aa-d41a-463a-acf6-79c6b2705f9c" (UID: "ae2706aa-d41a-463a-acf6-79c6b2705f9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.929135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.923787    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-token" pod "ae2706aa-d41a-463a-acf6-79c6b2705f9c" (UID: "ae2706aa-d41a-463a-acf6-79c6b2705f9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:55.929135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.926275    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f71c73-0d76-4483-b204-595f241c0a89-istio-token" pod "06f71c73-0d76-4483-b204-595f241c0a89" (UID: "06f71c73-0d76-4483-b204-595f241c0a89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:55.929135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.928819    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f71c73-0d76-4483-b204-595f241c0a89-istiod-ca-cert" pod "06f71c73-0d76-4483-b204-595f241c0a89" (UID: "06f71c73-0d76-4483-b204-595f241c0a89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:01:55.929865 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.929833    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-credential-socket" pod "06f71c73-0d76-4483-b204-595f241c0a89" (UID: "06f71c73-0d76-4483-b204-595f241c0a89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:55.943657 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.943630    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 10:01:55.948466 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.948445    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 10:01:55.998130 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998096    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-credential-socket\") pod \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") "
Nov 02 10:01:55.998297 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998152    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-workload-socket\") pod \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") "
Nov 02 10:01:55.998297 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998187    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-kube-api-access-tz4k9\" (UniqueName: \"kubernetes.io/projected/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-kube-api-access-tz4k9\") pod \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") "
Nov 02 10:01:55.998297 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998243    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-workload-certs\") pod \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") "
Nov 02 10:01:55.998297 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998287    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-workload-certs\") pod \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") "
Nov 02 10:01:55.998544 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998313    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istiod-ca-cert\") pod \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") "
Nov 02 10:01:55.998544 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998369    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-token\" (UniqueName: \"kubernetes.io/projected/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-token\") pod \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") "
Nov 02 10:01:55.998544 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998411    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-data\") pod \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") "
Nov 02 10:01:55.998544 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998446    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-credential-socket\") pod \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") "
Nov 02 10:01:55.998544 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998473    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-data\") pod \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") "
Nov 02 10:01:55.998544 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998507    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-token\" (UniqueName: \"kubernetes.io/projected/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-token\") pod \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") "
Nov 02 10:01:55.998544 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998536    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/041c3aa0-45b9-42f1-a86c-0cd1e4404593-kube-api-access-w2rrm\" (UniqueName: \"kubernetes.io/projected/041c3aa0-45b9-42f1-a86c-0cd1e4404593-kube-api-access-w2rrm\") pod \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") "
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998573    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-podinfo\") pod \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") "
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998614    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-podinfo\") pod \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") "
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998658    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-workload-socket\") pod \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") "
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998692    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istiod-ca-cert\") pod \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") "
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998779    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-envoy\") pod \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\" (UID: \"041c3aa0-45b9-42f1-a86c-0cd1e4404593\") "
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.998807    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-envoy\") pod \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\" (UID: \"58f3dc8a-8a94-48b8-b09f-41cf77f122f5\") "
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999036    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-credential-socket" pod "041c3aa0-45b9-42f1-a86c-0cd1e4404593" (UID: "041c3aa0-45b9-42f1-a86c-0cd1e4404593"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999201    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999216    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999226    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999235    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4j7z\" (UniqueName: \"kubernetes.io/projected/ae2706aa-d41a-463a-acf6-79c6b2705f9c-kube-api-access-m4j7z\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999245    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999254    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999262    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999271    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999281    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f71c73-0d76-4483-b204-595f241c0a89-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999294    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxwtc\" (UniqueName: \"kubernetes.io/projected/06f71c73-0d76-4483-b204-595f241c0a89-kube-api-access-hxwtc\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999302    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999312    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2706aa-d41a-463a-acf6-79c6b2705f9c-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999325    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f71c73-0d76-4483-b204-595f241c0a89-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999338    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999363    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999379    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f71c73-0d76-4483-b204-595f241c0a89-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999392    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999408    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f71c73-0d76-4483-b204-595f241c0a89-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999421    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2706aa-d41a-463a-acf6-79c6b2705f9c-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999639    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-credential-socket" pod "58f3dc8a-8a94-48b8-b09f-41cf77f122f5" (UID: "58f3dc8a-8a94-48b8-b09f-41cf77f122f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:55.999825    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-workload-socket" pod "041c3aa0-45b9-42f1-a86c-0cd1e4404593" (UID: "041c3aa0-45b9-42f1-a86c-0cd1e4404593"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.001568    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-envoy" pod "58f3dc8a-8a94-48b8-b09f-41cf77f122f5" (UID: "58f3dc8a-8a94-48b8-b09f-41cf77f122f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.001871    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istiod-ca-cert" pod "041c3aa0-45b9-42f1-a86c-0cd1e4404593" (UID: "041c3aa0-45b9-42f1-a86c-0cd1e4404593"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.001975    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-workload-certs" pod "041c3aa0-45b9-42f1-a86c-0cd1e4404593" (UID: "041c3aa0-45b9-42f1-a86c-0cd1e4404593"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.002063    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-workload-certs" pod "58f3dc8a-8a94-48b8-b09f-41cf77f122f5" (UID: "58f3dc8a-8a94-48b8-b09f-41cf77f122f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.002155    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-workload-socket" pod "58f3dc8a-8a94-48b8-b09f-41cf77f122f5" (UID: "58f3dc8a-8a94-48b8-b09f-41cf77f122f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.002286    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-data" pod "041c3aa0-45b9-42f1-a86c-0cd1e4404593" (UID: "041c3aa0-45b9-42f1-a86c-0cd1e4404593"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.002414    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istiod-ca-cert" pod "58f3dc8a-8a94-48b8-b09f-41cf77f122f5" (UID: "58f3dc8a-8a94-48b8-b09f-41cf77f122f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.002170    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-data" pod "58f3dc8a-8a94-48b8-b09f-41cf77f122f5" (UID: "58f3dc8a-8a94-48b8-b09f-41cf77f122f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:56.003032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.002814    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-token" pod "58f3dc8a-8a94-48b8-b09f-41cf77f122f5" (UID: "58f3dc8a-8a94-48b8-b09f-41cf77f122f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:56.004318 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.003054    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-podinfo" pod "041c3aa0-45b9-42f1-a86c-0cd1e4404593" (UID: "041c3aa0-45b9-42f1-a86c-0cd1e4404593"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:01:56.004318 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.004118    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-token" pod "041c3aa0-45b9-42f1-a86c-0cd1e4404593" (UID: "041c3aa0-45b9-42f1-a86c-0cd1e4404593"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:56.004513 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.004493    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-envoy" pod "041c3aa0-45b9-42f1-a86c-0cd1e4404593" (UID: "041c3aa0-45b9-42f1-a86c-0cd1e4404593"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:01:56.004749 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.004728    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-kube-api-access-tz4k9" pod "58f3dc8a-8a94-48b8-b09f-41cf77f122f5" (UID: "58f3dc8a-8a94-48b8-b09f-41cf77f122f5"). InnerVolumeSpecName "kube-api-access-tz4k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:56.006498 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.006475    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041c3aa0-45b9-42f1-a86c-0cd1e4404593-kube-api-access-w2rrm" pod "041c3aa0-45b9-42f1-a86c-0cd1e4404593" (UID: "041c3aa0-45b9-42f1-a86c-0cd1e4404593"). InnerVolumeSpecName "kube-api-access-w2rrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:01:56.006580 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.006510    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-podinfo" pod "58f3dc8a-8a94-48b8-b09f-41cf77f122f5" (UID: "58f3dc8a-8a94-48b8-b09f-41cf77f122f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:01:56.100447 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100341    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100447 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100405    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100447 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100420    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100447 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100434    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100447 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100449    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz4k9\" (UniqueName: \"kubernetes.io/projected/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-kube-api-access-tz4k9\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100722 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100462    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100722 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100475    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100722 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100486    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100722 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100498    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100722 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100509    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100722 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100523    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100722 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100535    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100722 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100547    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2rrm\" (UniqueName: \"kubernetes.io/projected/041c3aa0-45b9-42f1-a86c-0cd1e4404593-kube-api-access-w2rrm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100722 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100561    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100722 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100573    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100722 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100585    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f3dc8a-8a94-48b8-b09f-41cf77f122f5-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.100722 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.100598    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041c3aa0-45b9-42f1-a86c-0cd1e4404593-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:01:56.595254 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.595220    4614 generic.go:356] "Generic (PLEG): container finished" podID="06f71c73-0d76-4483-b204-595f241c0a89" containerID="c7fe0aa36bdc05573a1ef1e762b466a297a4aaa6081e7d0721da5f5f7480f0ed" exitCode=0
Nov 02 10:01:56.595456 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.595274    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"
Nov 02 10:01:56.595456 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.595299    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh" event={"ID":"06f71c73-0d76-4483-b204-595f241c0a89","Type":"ContainerDied","Data":"c7fe0aa36bdc05573a1ef1e762b466a297a4aaa6081e7d0721da5f5f7480f0ed"}
Nov 02 10:01:56.595456 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.595371    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh" event={"ID":"06f71c73-0d76-4483-b204-595f241c0a89","Type":"ContainerDied","Data":"f40ababa2b9fb22b537b3e4e0d6abf4f5f5117ed4f7f01ce9387622a594cae14"}
Nov 02 10:01:56.595456 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.595402    4614 scope.go:122] "RemoveContainer" containerID="c7fe0aa36bdc05573a1ef1e762b466a297a4aaa6081e7d0721da5f5f7480f0ed"
Nov 02 10:01:56.596932 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.596913    4614 generic.go:356] "Generic (PLEG): container finished" podID="ae2706aa-d41a-463a-acf6-79c6b2705f9c" containerID="48f208575364f300d286b90bb0c7622665ffb21b9a3df7fc159bbad69c181246" exitCode=0
Nov 02 10:01:56.597045 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.596962    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"
Nov 02 10:01:56.597045 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.597001    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492" event={"ID":"ae2706aa-d41a-463a-acf6-79c6b2705f9c","Type":"ContainerDied","Data":"48f208575364f300d286b90bb0c7622665ffb21b9a3df7fc159bbad69c181246"}
Nov 02 10:01:56.597160 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.597062    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492" event={"ID":"ae2706aa-d41a-463a-acf6-79c6b2705f9c","Type":"ContainerDied","Data":"ccb2a73cc7479c43b9c786d8b93792782506aa4778b6078a8942cd18feac59bb"}
Nov 02 10:01:56.598478 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.598453    4614 generic.go:356] "Generic (PLEG): container finished" podID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerID="7ddb4b3487498c3ca60b67429bc4fc1b60f9427141f4022f4c4cc3701778086c" exitCode=0
Nov 02 10:01:56.598589 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.598481    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" event={"ID":"58f3dc8a-8a94-48b8-b09f-41cf77f122f5","Type":"ContainerDied","Data":"7ddb4b3487498c3ca60b67429bc4fc1b60f9427141f4022f4c4cc3701778086c"}
Nov 02 10:01:56.598589 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.598515    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"
Nov 02 10:01:56.598589 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.598527    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk" event={"ID":"58f3dc8a-8a94-48b8-b09f-41cf77f122f5","Type":"ContainerDied","Data":"4eef4fa69e8b96601d2e097de9753a403754818b6380e9a1ea301fd6278e672b"}
Nov 02 10:01:56.603805 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.603782    4614 generic.go:356] "Generic (PLEG): container finished" podID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" containerID="c27db4bf0858e319748be554a041cef4a225f4879d5dc35f01f26e6754f0bd38" exitCode=0
Nov 02 10:01:56.603922 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.603833    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"
Nov 02 10:01:56.604091 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.603836    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" event={"ID":"041c3aa0-45b9-42f1-a86c-0cd1e4404593","Type":"ContainerDied","Data":"c27db4bf0858e319748be554a041cef4a225f4879d5dc35f01f26e6754f0bd38"}
Nov 02 10:01:56.604215 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.604114    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4" event={"ID":"041c3aa0-45b9-42f1-a86c-0cd1e4404593","Type":"ContainerDied","Data":"e7e1d3a99e72d222f846ea9ca6042dde0139934e8897ddcd95a6fad44ed2372f"}
Nov 02 10:01:56.606047 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.606032    4614 scope.go:122] "RemoveContainer" containerID="c7fe0aa36bdc05573a1ef1e762b466a297a4aaa6081e7d0721da5f5f7480f0ed"
Nov 02 10:01:56.606310 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:01:56.606288    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7fe0aa36bdc05573a1ef1e762b466a297a4aaa6081e7d0721da5f5f7480f0ed\": container with ID starting with c7fe0aa36bdc05573a1ef1e762b466a297a4aaa6081e7d0721da5f5f7480f0ed not found: ID does not exist" containerID="c7fe0aa36bdc05573a1ef1e762b466a297a4aaa6081e7d0721da5f5f7480f0ed"
Nov 02 10:01:56.606418 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.606321    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7fe0aa36bdc05573a1ef1e762b466a297a4aaa6081e7d0721da5f5f7480f0ed"} err="failed to get container status \"c7fe0aa36bdc05573a1ef1e762b466a297a4aaa6081e7d0721da5f5f7480f0ed\": rpc error: code = NotFound desc = could not find container \"c7fe0aa36bdc05573a1ef1e762b466a297a4aaa6081e7d0721da5f5f7480f0ed\": container with ID starting with c7fe0aa36bdc05573a1ef1e762b466a297a4aaa6081e7d0721da5f5f7480f0ed not found: ID does not exist"
Nov 02 10:01:56.606418 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.606343    4614 scope.go:122] "RemoveContainer" containerID="48f208575364f300d286b90bb0c7622665ffb21b9a3df7fc159bbad69c181246"
Nov 02 10:01:56.616955 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.616935    4614 scope.go:122] "RemoveContainer" containerID="48f208575364f300d286b90bb0c7622665ffb21b9a3df7fc159bbad69c181246"
Nov 02 10:01:56.617202 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:01:56.617185    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f208575364f300d286b90bb0c7622665ffb21b9a3df7fc159bbad69c181246\": container with ID starting with 48f208575364f300d286b90bb0c7622665ffb21b9a3df7fc159bbad69c181246 not found: ID does not exist" containerID="48f208575364f300d286b90bb0c7622665ffb21b9a3df7fc159bbad69c181246"
Nov 02 10:01:56.617260 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.617208    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f208575364f300d286b90bb0c7622665ffb21b9a3df7fc159bbad69c181246"} err="failed to get container status \"48f208575364f300d286b90bb0c7622665ffb21b9a3df7fc159bbad69c181246\": rpc error: code = NotFound desc = could not find container \"48f208575364f300d286b90bb0c7622665ffb21b9a3df7fc159bbad69c181246\": container with ID starting with 48f208575364f300d286b90bb0c7622665ffb21b9a3df7fc159bbad69c181246 not found: ID does not exist"
Nov 02 10:01:56.617260 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.617226    4614 scope.go:122] "RemoveContainer" containerID="7ddb4b3487498c3ca60b67429bc4fc1b60f9427141f4022f4c4cc3701778086c"
Nov 02 10:01:56.625168 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.625149    4614 scope.go:122] "RemoveContainer" containerID="7ddb4b3487498c3ca60b67429bc4fc1b60f9427141f4022f4c4cc3701778086c"
Nov 02 10:01:56.625436 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:01:56.625419    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ddb4b3487498c3ca60b67429bc4fc1b60f9427141f4022f4c4cc3701778086c\": container with ID starting with 7ddb4b3487498c3ca60b67429bc4fc1b60f9427141f4022f4c4cc3701778086c not found: ID does not exist" containerID="7ddb4b3487498c3ca60b67429bc4fc1b60f9427141f4022f4c4cc3701778086c"
Nov 02 10:01:56.625488 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.625442    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ddb4b3487498c3ca60b67429bc4fc1b60f9427141f4022f4c4cc3701778086c"} err="failed to get container status \"7ddb4b3487498c3ca60b67429bc4fc1b60f9427141f4022f4c4cc3701778086c\": rpc error: code = NotFound desc = could not find container \"7ddb4b3487498c3ca60b67429bc4fc1b60f9427141f4022f4c4cc3701778086c\": container with ID starting with 7ddb4b3487498c3ca60b67429bc4fc1b60f9427141f4022f4c4cc3701778086c not found: ID does not exist"
Nov 02 10:01:56.625488 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.625460    4614 scope.go:122] "RemoveContainer" containerID="c27db4bf0858e319748be554a041cef4a225f4879d5dc35f01f26e6754f0bd38"
Nov 02 10:01:56.630864 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.630845    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"]
Nov 02 10:01:56.634516 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.634499    4614 scope.go:122] "RemoveContainer" containerID="c27db4bf0858e319748be554a041cef4a225f4879d5dc35f01f26e6754f0bd38"
Nov 02 10:01:56.634770 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:01:56.634754    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27db4bf0858e319748be554a041cef4a225f4879d5dc35f01f26e6754f0bd38\": container with ID starting with c27db4bf0858e319748be554a041cef4a225f4879d5dc35f01f26e6754f0bd38 not found: ID does not exist" containerID="c27db4bf0858e319748be554a041cef4a225f4879d5dc35f01f26e6754f0bd38"
Nov 02 10:01:56.634837 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.634776    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27db4bf0858e319748be554a041cef4a225f4879d5dc35f01f26e6754f0bd38"} err="failed to get container status \"c27db4bf0858e319748be554a041cef4a225f4879d5dc35f01f26e6754f0bd38\": rpc error: code = NotFound desc = could not find container \"c27db4bf0858e319748be554a041cef4a225f4879d5dc35f01f26e6754f0bd38\": container with ID starting with c27db4bf0858e319748be554a041cef4a225f4879d5dc35f01f26e6754f0bd38 not found: ID does not exist"
Nov 02 10:01:56.635142 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.635105    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-58hgh"]
Nov 02 10:01:56.646344 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.646323    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"]
Nov 02 10:01:56.650171 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.650151    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-wmdl4"]
Nov 02 10:01:56.661669 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.661650    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"]
Nov 02 10:01:56.665988 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.665970    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-42xxk"]
Nov 02 10:01:56.678539 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.678520    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"]
Nov 02 10:01:56.684056 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:56.684037    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a110ef-openshift-default-754f5584f9-2d492"]
Nov 02 10:01:57.314683 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:57.314652    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" path="/var/lib/kubelet/pods/041c3aa0-45b9-42f1-a86c-0cd1e4404593/volumes"
Nov 02 10:01:57.315118 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:57.315105    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f71c73-0d76-4483-b204-595f241c0a89" path="/var/lib/kubelet/pods/06f71c73-0d76-4483-b204-595f241c0a89/volumes"
Nov 02 10:01:57.315547 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:57.315534    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" path="/var/lib/kubelet/pods/58f3dc8a-8a94-48b8-b09f-41cf77f122f5/volumes"
Nov 02 10:01:57.316254 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:01:57.316240    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2706aa-d41a-463a-acf6-79c6b2705f9c" path="/var/lib/kubelet/pods/ae2706aa-d41a-463a-acf6-79c6b2705f9c/volumes"
Nov 02 10:02:09.638561 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:02:09.638522    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:02:24.619572 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.619533    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"]
Nov 02 10:02:24.620211 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.620128    4614 cpu_manager.go:414] "containerMap: removing container" podUID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" containerName="istio-proxy"
Nov 02 10:02:24.620211 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.620144    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" containerName="istio-proxy"
Nov 02 10:02:24.620211 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.620178    4614 cpu_manager.go:414] "containerMap: removing container" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerName="istio-proxy"
Nov 02 10:02:24.620211 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.620186    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerName="istio-proxy"
Nov 02 10:02:24.620211 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.620208    4614 cpu_manager.go:414] "containerMap: removing container" podUID="06f71c73-0d76-4483-b204-595f241c0a89" containerName="istio-proxy"
Nov 02 10:02:24.620498 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.620217    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f71c73-0d76-4483-b204-595f241c0a89" containerName="istio-proxy"
Nov 02 10:02:24.620498 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.620227    4614 cpu_manager.go:414] "containerMap: removing container" podUID="ae2706aa-d41a-463a-acf6-79c6b2705f9c" containerName="istio-proxy"
Nov 02 10:02:24.620498 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.620234    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2706aa-d41a-463a-acf6-79c6b2705f9c" containerName="istio-proxy"
Nov 02 10:02:24.620498 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.620321    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f71c73-0d76-4483-b204-595f241c0a89" containerName="istio-proxy"
Nov 02 10:02:24.620498 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.620331    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2706aa-d41a-463a-acf6-79c6b2705f9c" containerName="istio-proxy"
Nov 02 10:02:24.620498 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.620345    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="041c3aa0-45b9-42f1-a86c-0cd1e4404593" containerName="istio-proxy"
Nov 02 10:02:24.620498 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.620377    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f3dc8a-8a94-48b8-b09f-41cf77f122f5" containerName="istio-proxy"
Nov 02 10:02:24.625640 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.625615    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:24.632297 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.632245    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-d1ad6b3bd8a0\""
Nov 02 10:02:24.635008 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.634983    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"]
Nov 02 10:02:24.719872 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.719835    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd2bs\" (UniqueName: \"kubernetes.io/projected/433b0012-a699-4536-a0e0-e090a67feb6e-kube-api-access-hd2bs\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:24.720042 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.719965    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433b0012-a699-4536-a0e0-e090a67feb6e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:24.720042 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.719993    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/433b0012-a699-4536-a0e0-e090a67feb6e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:24.720042 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.720024    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/433b0012-a699-4536-a0e0-e090a67feb6e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:24.820710 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.820674    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433b0012-a699-4536-a0e0-e090a67feb6e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:24.820876 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.820717    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/433b0012-a699-4536-a0e0-e090a67feb6e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:24.820876 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.820754    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/433b0012-a699-4536-a0e0-e090a67feb6e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:24.820876 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.820789    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd2bs\" (UniqueName: \"kubernetes.io/projected/433b0012-a699-4536-a0e0-e090a67feb6e-kube-api-access-hd2bs\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:24.821521 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.821494    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/433b0012-a699-4536-a0e0-e090a67feb6e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:24.821660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.821525    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433b0012-a699-4536-a0e0-e090a67feb6e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:24.823332 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.823311    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/433b0012-a699-4536-a0e0-e090a67feb6e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:24.833279 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.833259    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd2bs\" (UniqueName: \"kubernetes.io/projected/433b0012-a699-4536-a0e0-e090a67feb6e-kube-api-access-hd2bs\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:24.942231 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.942152    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:24.962016 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.961987    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"]
Nov 02 10:02:24.967339 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.967309    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:24.972188 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.972163    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-6ce0a95c3878\""
Nov 02 10:02:24.975951 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:24.975927    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"]
Nov 02 10:02:25.022175 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.022143    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9xns\" (UniqueName: \"kubernetes.io/projected/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-kube-api-access-v9xns\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:25.022466 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.022439    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:25.022624 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.022606    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:25.022840 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.022820    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:25.123910 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.123870    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9xns\" (UniqueName: \"kubernetes.io/projected/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-kube-api-access-v9xns\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:25.124111 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.123943    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:25.124111 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.123972    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:25.124111 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.124066    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:25.124656 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.124634    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:25.124751 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.124733    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:25.126752 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.126726    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:25.136707 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.136685    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9xns\" (UniqueName: \"kubernetes.io/projected/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-kube-api-access-v9xns\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:25.224310 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.224283    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"]
Nov 02 10:02:25.226122 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:02:25.226090    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433b0012_a699_4536_a0e0_e090a67feb6e.slice/crio-5999ac2d0447e3f1fa334ea5749b7b0b8500a6cfb51da3d08a1d08c910acf443 WatchSource:0}: Error finding container 5999ac2d0447e3f1fa334ea5749b7b0b8500a6cfb51da3d08a1d08c910acf443: Status 404 returned error can't find the container with id 5999ac2d0447e3f1fa334ea5749b7b0b8500a6cfb51da3d08a1d08c910acf443
Nov 02 10:02:25.286420 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.286396    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:25.561675 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.561650    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"]
Nov 02 10:02:25.563923 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:02:25.563892    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1fcffb2_7d6b_4c9d_a563_fab56f5eaaa5.slice/crio-4519871d19ecd8afc4e6e97f6b049e9934c4b2aa9357844a3c4a9dc79ad651a2 WatchSource:0}: Error finding container 4519871d19ecd8afc4e6e97f6b049e9934c4b2aa9357844a3c4a9dc79ad651a2: Status 404 returned error can't find the container with id 4519871d19ecd8afc4e6e97f6b049e9934c4b2aa9357844a3c4a9dc79ad651a2
Nov 02 10:02:25.762088 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.762047    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6" event={"ID":"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5","Type":"ContainerStarted","Data":"d5f867723ae7c93df05299d1496120b66bd8b6996aa856ce58558b079216d5c2"}
Nov 02 10:02:25.762595 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.762104    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6" event={"ID":"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5","Type":"ContainerStarted","Data":"4519871d19ecd8afc4e6e97f6b049e9934c4b2aa9357844a3c4a9dc79ad651a2"}
Nov 02 10:02:25.762595 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.762213    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:25.763827 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.763796    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7" event={"ID":"433b0012-a699-4536-a0e0-e090a67feb6e","Type":"ContainerStarted","Data":"9ffa44c2bea0756f0b86a00cb2f2ed3ad50dfc0e823350202f1ff3898fbe763b"}
Nov 02 10:02:25.763971 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.763835    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7" event={"ID":"433b0012-a699-4536-a0e0-e090a67feb6e","Type":"ContainerStarted","Data":"5999ac2d0447e3f1fa334ea5749b7b0b8500a6cfb51da3d08a1d08c910acf443"}
Nov 02 10:02:25.763971 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.763946    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:25.781272 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.781220    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6" podStartSLOduration=1.7812075219999999 podStartE2EDuration="1.781207522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:02:25.779886264 +0000 UTC m=+1626.612544346" watchObservedRunningTime="2026-11-02 10:02:25.781207522 +0000 UTC m=+1626.613865629"
Nov 02 10:02:25.800520 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:25.800469    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7" podStartSLOduration=1.800456748 podStartE2EDuration="1.800456748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:02:25.797886442 +0000 UTC m=+1626.630544545" watchObservedRunningTime="2026-11-02 10:02:25.800456748 +0000 UTC m=+1626.633114857"
Nov 02 10:02:36.777592 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:36.777564    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:02:36.778047 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:36.777639    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:02:36.836811 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:36.836779    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"]
Nov 02 10:02:36.849269 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:36.849240    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"]
Nov 02 10:02:39.635423 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:02:39.635385    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:02:41.837787 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:41.837741    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8" podUID="bddbc2f2-63b0-4267-8bf9-b54ab2d1b874" containerName="payload-processing" containerID="cri-o://ed08e2e2889abe87ab235c5569ce622bd12c86950479961b7b2a16cec204dabd" gracePeriod=25
Nov 02 10:02:41.850277 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:41.850243    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj" podUID="35561b22-9eca-405a-b437-e0ad2586d797" containerName="payload-processing" containerID="cri-o://290ec5df2b522a5d7a088177c76eeb933c7fb18f8b0f67d752a88fe1d6f72aa2" gracePeriod=25
Nov 02 10:02:42.107075 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.107052    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:02:42.137218 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.137189    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:02:42.226884 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.226853    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35561b22-9eca-405a-b437-e0ad2586d797-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35561b22-9eca-405a-b437-e0ad2586d797-plugins-config-volume\") pod \"35561b22-9eca-405a-b437-e0ad2586d797\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") "
Nov 02 10:02:42.227066 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.226929    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-kube-api-access-g9z5w\" (UniqueName: \"kubernetes.io/projected/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-kube-api-access-g9z5w\") pod \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") "
Nov 02 10:02:42.227066 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.226976    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-provider-credentials\") pod \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") "
Nov 02 10:02:42.227066 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.226996    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-routing-overlay-volume\") pod \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") "
Nov 02 10:02:42.227066 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.227015    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-plugins-config-volume\") pod \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\" (UID: \"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874\") "
Nov 02 10:02:42.227306 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.227151    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35561b22-9eca-405a-b437-e0ad2586d797-kube-api-access-m85jn\" (UniqueName: \"kubernetes.io/projected/35561b22-9eca-405a-b437-e0ad2586d797-kube-api-access-m85jn\") pod \"35561b22-9eca-405a-b437-e0ad2586d797\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") "
Nov 02 10:02:42.227306 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.227200    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35561b22-9eca-405a-b437-e0ad2586d797-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35561b22-9eca-405a-b437-e0ad2586d797-routing-overlay-volume\") pod \"35561b22-9eca-405a-b437-e0ad2586d797\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") "
Nov 02 10:02:42.227306 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.227228    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35561b22-9eca-405a-b437-e0ad2586d797-provider-credentials\" (UniqueName: \"kubernetes.io/projected/35561b22-9eca-405a-b437-e0ad2586d797-provider-credentials\") pod \"35561b22-9eca-405a-b437-e0ad2586d797\" (UID: \"35561b22-9eca-405a-b437-e0ad2586d797\") "
Nov 02 10:02:42.227543 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.227434    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35561b22-9eca-405a-b437-e0ad2586d797-plugins-config-volume" pod "35561b22-9eca-405a-b437-e0ad2586d797" (UID: "35561b22-9eca-405a-b437-e0ad2586d797"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:02:42.227649 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.227617    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35561b22-9eca-405a-b437-e0ad2586d797-routing-overlay-volume" pod "35561b22-9eca-405a-b437-e0ad2586d797" (UID: "35561b22-9eca-405a-b437-e0ad2586d797"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:02:42.227730 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.227679    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35561b22-9eca-405a-b437-e0ad2586d797-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:02:42.227730 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.227698    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-routing-overlay-volume" pod "bddbc2f2-63b0-4267-8bf9-b54ab2d1b874" (UID: "bddbc2f2-63b0-4267-8bf9-b54ab2d1b874"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:02:42.227853 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.227742    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-plugins-config-volume" pod "bddbc2f2-63b0-4267-8bf9-b54ab2d1b874" (UID: "bddbc2f2-63b0-4267-8bf9-b54ab2d1b874"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:02:42.229339 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.229315    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-kube-api-access-g9z5w" pod "bddbc2f2-63b0-4267-8bf9-b54ab2d1b874" (UID: "bddbc2f2-63b0-4267-8bf9-b54ab2d1b874"). InnerVolumeSpecName "kube-api-access-g9z5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:02:42.229480 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.229459    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35561b22-9eca-405a-b437-e0ad2586d797-provider-credentials" pod "35561b22-9eca-405a-b437-e0ad2586d797" (UID: "35561b22-9eca-405a-b437-e0ad2586d797"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:02:42.229661 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.229639    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-provider-credentials" pod "bddbc2f2-63b0-4267-8bf9-b54ab2d1b874" (UID: "bddbc2f2-63b0-4267-8bf9-b54ab2d1b874"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:02:42.229768 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.229751    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35561b22-9eca-405a-b437-e0ad2586d797-kube-api-access-m85jn" pod "35561b22-9eca-405a-b437-e0ad2586d797" (UID: "35561b22-9eca-405a-b437-e0ad2586d797"). InnerVolumeSpecName "kube-api-access-m85jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:02:42.329210 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.329174    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9z5w\" (UniqueName: \"kubernetes.io/projected/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-kube-api-access-g9z5w\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:02:42.329210 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.329204    4614 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-provider-credentials\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:02:42.329210 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.329215    4614 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-routing-overlay-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:02:42.329476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.329223    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:02:42.329476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.329234    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m85jn\" (UniqueName: \"kubernetes.io/projected/35561b22-9eca-405a-b437-e0ad2586d797-kube-api-access-m85jn\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:02:42.329476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.329248    4614 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35561b22-9eca-405a-b437-e0ad2586d797-routing-overlay-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:02:42.329476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.329256    4614 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35561b22-9eca-405a-b437-e0ad2586d797-provider-credentials\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:02:42.869149 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.869116    4614 generic.go:356] "Generic (PLEG): container finished" podID="35561b22-9eca-405a-b437-e0ad2586d797" containerID="290ec5df2b522a5d7a088177c76eeb933c7fb18f8b0f67d752a88fe1d6f72aa2" exitCode=0
Nov 02 10:02:42.869558 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.869169    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"
Nov 02 10:02:42.869558 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.869207    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj" event={"ID":"35561b22-9eca-405a-b437-e0ad2586d797","Type":"ContainerDied","Data":"290ec5df2b522a5d7a088177c76eeb933c7fb18f8b0f67d752a88fe1d6f72aa2"}
Nov 02 10:02:42.869558 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.869262    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj" event={"ID":"35561b22-9eca-405a-b437-e0ad2586d797","Type":"ContainerDied","Data":"1a22b9191ba6163fe62b1a4ce4f87b72affd304a23361d1b7478ba72ab7ef05e"}
Nov 02 10:02:42.869558 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.869290    4614 scope.go:122] "RemoveContainer" containerID="290ec5df2b522a5d7a088177c76eeb933c7fb18f8b0f67d752a88fe1d6f72aa2"
Nov 02 10:02:42.870655 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.870634    4614 generic.go:356] "Generic (PLEG): container finished" podID="bddbc2f2-63b0-4267-8bf9-b54ab2d1b874" containerID="ed08e2e2889abe87ab235c5569ce622bd12c86950479961b7b2a16cec204dabd" exitCode=0
Nov 02 10:02:42.870730 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.870672    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"
Nov 02 10:02:42.870783 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.870745    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8" event={"ID":"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874","Type":"ContainerDied","Data":"ed08e2e2889abe87ab235c5569ce622bd12c86950479961b7b2a16cec204dabd"}
Nov 02 10:02:42.870821 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.870787    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8" event={"ID":"bddbc2f2-63b0-4267-8bf9-b54ab2d1b874","Type":"ContainerDied","Data":"728e787f402b2e5a840cc6197f7f8935dfb2498a8c97fe034173de849c5a5167"}
Nov 02 10:02:42.879697 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.879679    4614 scope.go:122] "RemoveContainer" containerID="290ec5df2b522a5d7a088177c76eeb933c7fb18f8b0f67d752a88fe1d6f72aa2"
Nov 02 10:02:42.879988 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:02:42.879969    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290ec5df2b522a5d7a088177c76eeb933c7fb18f8b0f67d752a88fe1d6f72aa2\": container with ID starting with 290ec5df2b522a5d7a088177c76eeb933c7fb18f8b0f67d752a88fe1d6f72aa2 not found: ID does not exist" containerID="290ec5df2b522a5d7a088177c76eeb933c7fb18f8b0f67d752a88fe1d6f72aa2"
Nov 02 10:02:42.880052 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.879998    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290ec5df2b522a5d7a088177c76eeb933c7fb18f8b0f67d752a88fe1d6f72aa2"} err="failed to get container status \"290ec5df2b522a5d7a088177c76eeb933c7fb18f8b0f67d752a88fe1d6f72aa2\": rpc error: code = NotFound desc = could not find container \"290ec5df2b522a5d7a088177c76eeb933c7fb18f8b0f67d752a88fe1d6f72aa2\": container with ID starting with 290ec5df2b522a5d7a088177c76eeb933c7fb18f8b0f67d752a88fe1d6f72aa2 not found: ID does not exist"
Nov 02 10:02:42.880052 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.880016    4614 scope.go:122] "RemoveContainer" containerID="ed08e2e2889abe87ab235c5569ce622bd12c86950479961b7b2a16cec204dabd"
Nov 02 10:02:42.887770 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.887755    4614 scope.go:122] "RemoveContainer" containerID="ed08e2e2889abe87ab235c5569ce622bd12c86950479961b7b2a16cec204dabd"
Nov 02 10:02:42.888000 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:02:42.887983    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed08e2e2889abe87ab235c5569ce622bd12c86950479961b7b2a16cec204dabd\": container with ID starting with ed08e2e2889abe87ab235c5569ce622bd12c86950479961b7b2a16cec204dabd not found: ID does not exist" containerID="ed08e2e2889abe87ab235c5569ce622bd12c86950479961b7b2a16cec204dabd"
Nov 02 10:02:42.888042 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.888006    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed08e2e2889abe87ab235c5569ce622bd12c86950479961b7b2a16cec204dabd"} err="failed to get container status \"ed08e2e2889abe87ab235c5569ce622bd12c86950479961b7b2a16cec204dabd\": rpc error: code = NotFound desc = could not find container \"ed08e2e2889abe87ab235c5569ce622bd12c86950479961b7b2a16cec204dabd\": container with ID starting with ed08e2e2889abe87ab235c5569ce622bd12c86950479961b7b2a16cec204dabd not found: ID does not exist"
Nov 02 10:02:42.893012 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.892990    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"]
Nov 02 10:02:42.905047 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.905028    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-7487czqfj"]
Nov 02 10:02:42.915929 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.915911    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"]
Nov 02 10:02:42.923779 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:42.923756    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-5bfbmpph8"]
Nov 02 10:02:43.314432 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:43.314399    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35561b22-9eca-405a-b437-e0ad2586d797" path="/var/lib/kubelet/pods/35561b22-9eca-405a-b437-e0ad2586d797/volumes"
Nov 02 10:02:43.314796 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:43.314783    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bddbc2f2-63b0-4267-8bf9-b54ab2d1b874" path="/var/lib/kubelet/pods/bddbc2f2-63b0-4267-8bf9-b54ab2d1b874/volumes"
Nov 02 10:02:47.278498 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.278457    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"]
Nov 02 10:02:47.279048 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.279027    4614 cpu_manager.go:414] "containerMap: removing container" podUID="35561b22-9eca-405a-b437-e0ad2586d797" containerName="payload-processing"
Nov 02 10:02:47.279129 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.279051    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35561b22-9eca-405a-b437-e0ad2586d797" containerName="payload-processing"
Nov 02 10:02:47.279129 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.279080    4614 cpu_manager.go:414] "containerMap: removing container" podUID="bddbc2f2-63b0-4267-8bf9-b54ab2d1b874" containerName="payload-processing"
Nov 02 10:02:47.279129 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.279090    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bddbc2f2-63b0-4267-8bf9-b54ab2d1b874" containerName="payload-processing"
Nov 02 10:02:47.279289 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.279215    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="35561b22-9eca-405a-b437-e0ad2586d797" containerName="payload-processing"
Nov 02 10:02:47.279289 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.279230    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="bddbc2f2-63b0-4267-8bf9-b54ab2d1b874" containerName="payload-processing"
Nov 02 10:02:47.283246 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.283223    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:47.293002 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.292983    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-407717c0a9b4\""
Nov 02 10:02:47.295240 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.295214    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"]
Nov 02 10:02:47.384561 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.384524    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:47.384759 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.384574    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:47.384759 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.384677    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpfss\" (UniqueName: \"kubernetes.io/projected/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-kube-api-access-lpfss\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:47.384759 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.384718    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:47.485920 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.485882    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:47.486099 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.485931    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:47.486099 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.485977    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpfss\" (UniqueName: \"kubernetes.io/projected/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-kube-api-access-lpfss\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:47.486099 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.485995    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:47.486680 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.486647    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:47.486788 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.486769    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:47.488626 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.488605    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:47.505489 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.505404    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpfss\" (UniqueName: \"kubernetes.io/projected/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-kube-api-access-lpfss\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:47.596744 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.596666    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:47.619532 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.619499    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"]
Nov 02 10:02:47.623888 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.623870    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:47.630540 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.630515    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-4c0df7e08666\""
Nov 02 10:02:47.633524 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.633502    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"]
Nov 02 10:02:47.788444 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.788408    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5hwn\" (UniqueName: \"kubernetes.io/projected/ef7ef707-ea6c-425b-89db-ff48b54c55b9-kube-api-access-h5hwn\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796c85hlm\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:47.788618 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.788462    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7ef707-ea6c-425b-89db-ff48b54c55b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796c85hlm\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:47.788618 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.788540    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef7ef707-ea6c-425b-89db-ff48b54c55b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796c85hlm\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:47.788696 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.788629    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef7ef707-ea6c-425b-89db-ff48b54c55b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796c85hlm\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:47.890086 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.889974    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7ef707-ea6c-425b-89db-ff48b54c55b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796c85hlm\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:47.890086 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.890068    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef7ef707-ea6c-425b-89db-ff48b54c55b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796c85hlm\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:47.890312 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.890118    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef7ef707-ea6c-425b-89db-ff48b54c55b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796c85hlm\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:47.890312 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.890177    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5hwn\" (UniqueName: \"kubernetes.io/projected/ef7ef707-ea6c-425b-89db-ff48b54c55b9-kube-api-access-h5hwn\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796c85hlm\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:47.890782 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.890757    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef7ef707-ea6c-425b-89db-ff48b54c55b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796c85hlm\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:47.890883 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.890822    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7ef707-ea6c-425b-89db-ff48b54c55b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796c85hlm\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:47.892799 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.892777    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef7ef707-ea6c-425b-89db-ff48b54c55b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796c85hlm\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:47.897557 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.897533    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"]
Nov 02 10:02:47.900498 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:02:47.900474    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e02dfa_4344_4948_8c7a_4931fd8d4bf7.slice/crio-8bb0999729ea56330dfe77bf7b6502ab429b9bba5d449af77c9ed88c1e1f95d2 WatchSource:0}: Error finding container 8bb0999729ea56330dfe77bf7b6502ab429b9bba5d449af77c9ed88c1e1f95d2: Status 404 returned error can't find the container with id 8bb0999729ea56330dfe77bf7b6502ab429b9bba5d449af77c9ed88c1e1f95d2
Nov 02 10:02:47.914484 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.914467    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5hwn\" (UniqueName: \"kubernetes.io/projected/ef7ef707-ea6c-425b-89db-ff48b54c55b9-kube-api-access-h5hwn\") pod \"payload-processing-external-model-e2e-extmod-a-629129-796c85hlm\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:47.942723 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:47.942695    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:48.225053 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.225023    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"]
Nov 02 10:02:48.226464 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:02:48.226435    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7ef707_ea6c_425b_89db_ff48b54c55b9.slice/crio-d9011337567066a318ad1a1fc44ca25d20b81d14d8903f61927bdc76bc6b6ddc WatchSource:0}: Error finding container d9011337567066a318ad1a1fc44ca25d20b81d14d8903f61927bdc76bc6b6ddc: Status 404 returned error can't find the container with id d9011337567066a318ad1a1fc44ca25d20b81d14d8903f61927bdc76bc6b6ddc
Nov 02 10:02:48.666560 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.666479    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"]
Nov 02 10:02:48.670410 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.670389    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.687609 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.687584    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"]
Nov 02 10:02:48.800904 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.800870    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.800904 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.800906    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.801119 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.800936    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.801119 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.801069    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.801196 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.801117    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.801236 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.801198    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-699xx\" (UniqueName: \"kubernetes.io/projected/2e4b65a7-462d-4831-8cca-65f29dac08df-kube-api-access-699xx\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.801236 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.801224    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.801307 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.801246    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.801307 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.801262    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4b65a7-462d-4831-8cca-65f29dac08df-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.842923 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.842881    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"]
Nov 02 10:02:48.847409 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.847382    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"
Nov 02 10:02:48.853426 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.852679    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-c2f625\"/\"kube-root-ca.crt\""
Nov 02 10:02:48.853426 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.852750    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-c2f625\"/\"e2e-usage-sim-78b35d-cert\""
Nov 02 10:02:48.853426 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.852855    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-c2f625\"/\"openshift-service-ca.crt\""
Nov 02 10:02:48.853426 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.852947    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-c2f625\"/\"default-dockercfg-b7j6f\""
Nov 02 10:02:48.891310 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.886975    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"]
Nov 02 10:02:48.903372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.902877    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.903372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.902943    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-699xx\" (UniqueName: \"kubernetes.io/projected/2e4b65a7-462d-4831-8cca-65f29dac08df-kube-api-access-699xx\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.903372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.902980    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.903372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.903012    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.903372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.903037    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4b65a7-462d-4831-8cca-65f29dac08df-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.903372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.903095    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.903372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.903123    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.903372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.903154    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.903372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.903219    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.904210 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.903689    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-workload-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.904210 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.904038    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-workload-certs\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.906113 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.906074    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-credential-socket\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.906323 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.906305    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4b65a7-462d-4831-8cca-65f29dac08df-istiod-ca-cert\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.906645 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.906583    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-data\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.913842 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.913812    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-podinfo\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.927372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.916284    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-envoy\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.927372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.923249    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm" event={"ID":"ef7ef707-ea6c-425b-89db-ff48b54c55b9","Type":"ContainerStarted","Data":"7457de31622aea41e6c6e42416f6e2ac24968b3f91465d50ebce65c8ebfd407a"}
Nov 02 10:02:48.927372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.923308    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm" event={"ID":"ef7ef707-ea6c-425b-89db-ff48b54c55b9","Type":"ContainerStarted","Data":"d9011337567066a318ad1a1fc44ca25d20b81d14d8903f61927bdc76bc6b6ddc"}
Nov 02 10:02:48.928128 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.928096    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:48.930339 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.930310    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v" event={"ID":"11e02dfa-4344-4948-8c7a-4931fd8d4bf7","Type":"ContainerStarted","Data":"24dd152670aad1a41397a98bebbae461793357bd704c4afe99e0987667b0e721"}
Nov 02 10:02:48.930484 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.930398    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v" event={"ID":"11e02dfa-4344-4948-8c7a-4931fd8d4bf7","Type":"ContainerStarted","Data":"8bb0999729ea56330dfe77bf7b6502ab429b9bba5d449af77c9ed88c1e1f95d2"}
Nov 02 10:02:48.930484 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.930430    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:48.931303 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.931278    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-699xx\" (UniqueName: \"kubernetes.io/projected/2e4b65a7-462d-4831-8cca-65f29dac08df-kube-api-access-699xx\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.931454 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.931298    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-token\") pod \"e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") " pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:48.950537 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.950488    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm" podStartSLOduration=1.9504719289999999 podStartE2EDuration="1.950471929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:02:48.950377078 +0000 UTC m=+1649.783035162" watchObservedRunningTime="2026-11-02 10:02:48.950471929 +0000 UTC m=+1649.783130010"
Nov 02 10:02:48.976960 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.976915    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v" podStartSLOduration=1.976902755 podStartE2EDuration="1.976902755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:02:48.974561316 +0000 UTC m=+1649.807219399" watchObservedRunningTime="2026-11-02 10:02:48.976902755 +0000 UTC m=+1649.809560814"
Nov 02 10:02:48.990563 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:48.990533    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:49.004445 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.004407    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/91a9290d-00fc-4af7-a072-60c2206112c3-tls\") pod \"e2e-usage-sim-78b35d-59c776c7c7-gbsrn\" (UID: \"91a9290d-00fc-4af7-a072-60c2206112c3\") " pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"
Nov 02 10:02:49.004586 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.004566    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq756\" (UniqueName: \"kubernetes.io/projected/91a9290d-00fc-4af7-a072-60c2206112c3-kube-api-access-tq756\") pod \"e2e-usage-sim-78b35d-59c776c7c7-gbsrn\" (UID: \"91a9290d-00fc-4af7-a072-60c2206112c3\") " pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"
Nov 02 10:02:49.109426 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.107950    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/91a9290d-00fc-4af7-a072-60c2206112c3-tls\") pod \"e2e-usage-sim-78b35d-59c776c7c7-gbsrn\" (UID: \"91a9290d-00fc-4af7-a072-60c2206112c3\") " pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"
Nov 02 10:02:49.109426 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.108155    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq756\" (UniqueName: \"kubernetes.io/projected/91a9290d-00fc-4af7-a072-60c2206112c3-kube-api-access-tq756\") pod \"e2e-usage-sim-78b35d-59c776c7c7-gbsrn\" (UID: \"91a9290d-00fc-4af7-a072-60c2206112c3\") " pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"
Nov 02 10:02:49.118944 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.118909    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/91a9290d-00fc-4af7-a072-60c2206112c3-tls\") pod \"e2e-usage-sim-78b35d-59c776c7c7-gbsrn\" (UID: \"91a9290d-00fc-4af7-a072-60c2206112c3\") " pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"
Nov 02 10:02:49.121562 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.121518    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq756\" (UniqueName: \"kubernetes.io/projected/91a9290d-00fc-4af7-a072-60c2206112c3-kube-api-access-tq756\") pod \"e2e-usage-sim-78b35d-59c776c7c7-gbsrn\" (UID: \"91a9290d-00fc-4af7-a072-60c2206112c3\") " pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"
Nov 02 10:02:49.185664 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.185554    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"
Nov 02 10:02:49.293675 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.293640    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"]
Nov 02 10:02:49.297693 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:02:49.297591    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4b65a7_462d_4831_8cca_65f29dac08df.slice/crio-c60160da7df2a6bf2e07b83f8f7407389ad53ccaa0d605226ed2a4af60249f5b WatchSource:0}: Error finding container c60160da7df2a6bf2e07b83f8f7407389ad53ccaa0d605226ed2a4af60249f5b: Status 404 returned error can't find the container with id c60160da7df2a6bf2e07b83f8f7407389ad53ccaa0d605226ed2a4af60249f5b
Nov 02 10:02:49.300097 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.300066    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:02:49.300197 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.300130    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:02:49.300197 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.300158    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:02:49.478937 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.478902    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"]
Nov 02 10:02:49.482492 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:02:49.482464    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91a9290d_00fc_4af7_a072_60c2206112c3.slice/crio-e239d99bede6cf114cc82ddc63d47aa2c278a3a82aa3f77bc25cc0e5c441f026 WatchSource:0}: Error finding container e239d99bede6cf114cc82ddc63d47aa2c278a3a82aa3f77bc25cc0e5c441f026: Status 404 returned error can't find the container with id e239d99bede6cf114cc82ddc63d47aa2c278a3a82aa3f77bc25cc0e5c441f026
Nov 02 10:02:49.954158 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.954055    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx" event={"ID":"2e4b65a7-462d-4831-8cca-65f29dac08df","Type":"ContainerStarted","Data":"d11e1e11e374f42839eaa4e03c5b96e2a6e3d5ec2505793c0e6e81112b9177ac"}
Nov 02 10:02:49.954158 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.954123    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx" event={"ID":"2e4b65a7-462d-4831-8cca-65f29dac08df","Type":"ContainerStarted","Data":"c60160da7df2a6bf2e07b83f8f7407389ad53ccaa0d605226ed2a4af60249f5b"}
Nov 02 10:02:49.955892 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.955865    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn" event={"ID":"91a9290d-00fc-4af7-a072-60c2206112c3","Type":"ContainerStarted","Data":"840f4fd5a401d35036e92d2053810ed68ec93f4a00c869e665bf83bd04d85133"}
Nov 02 10:02:49.955892 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.955896    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn" event={"ID":"91a9290d-00fc-4af7-a072-60c2206112c3","Type":"ContainerStarted","Data":"e239d99bede6cf114cc82ddc63d47aa2c278a3a82aa3f77bc25cc0e5c441f026"}
Nov 02 10:02:49.956070 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.955994    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"
Nov 02 10:02:49.961520 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.961500    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"
Nov 02 10:02:49.977013 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.976974    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx" podStartSLOduration=1.976961811 podStartE2EDuration="1.976961811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:02:49.97572052 +0000 UTC m=+1650.808378601" watchObservedRunningTime="2026-11-02 10:02:49.976961811 +0000 UTC m=+1650.809619888"
Nov 02 10:02:49.991135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.991093    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:49.994300 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:49.994256    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn" podStartSLOduration=1.994243488 podStartE2EDuration="1.994243488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:02:49.993167336 +0000 UTC m=+1650.825825422" watchObservedRunningTime="2026-11-02 10:02:49.994243488 +0000 UTC m=+1650.826901564"
Nov 02 10:02:50.992960 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:50.992929    4614 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 10:02:50.993383 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:50.992980    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx" podUID="2e4b65a7-462d-4831-8cca-65f29dac08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 10:02:51.991170 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:51.991141    4614 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 10:02:51.991337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:51.991188    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx" podUID="2e4b65a7-462d-4831-8cca-65f29dac08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 10:02:52.991249 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:52.991215    4614 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 10:02:52.991664 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:52.991266    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx" podUID="2e4b65a7-462d-4831-8cca-65f29dac08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 10:02:53.991502 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:53.991465    4614 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 10:02:53.991973 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:53.991515    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx" podUID="2e4b65a7-462d-4831-8cca-65f29dac08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 10:02:54.991475 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:54.991439    4614 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 10:02:54.991665 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:54.991518    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx" podUID="2e4b65a7-462d-4831-8cca-65f29dac08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 10:02:55.254388 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:55.254279    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"]
Nov 02 10:02:55.258227 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:55.258199    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:02:55.582235 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:55.582149    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"]
Nov 02 10:02:55.585237 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:55.585182    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:02:55.995208 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:55.995179    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:55.995724 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:55.995469    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:02:55.996224 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:02:55.996204    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:03:00.255499 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.255394    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v" podUID="11e02dfa-4344-4948-8c7a-4931fd8d4bf7" containerName="payload-processing" containerID="cri-o://24dd152670aad1a41397a98bebbae461793357bd704c4afe99e0987667b0e721" gracePeriod=25
Nov 02 10:03:00.510998 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.510934    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:03:00.553379 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.553330    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-kube-api-access-lpfss\" (UniqueName: \"kubernetes.io/projected/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-kube-api-access-lpfss\") pod \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") "
Nov 02 10:03:00.553556 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.553403    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-provider-credentials\") pod \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") "
Nov 02 10:03:00.553655 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.553634    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-routing-overlay-volume\") pod \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") "
Nov 02 10:03:00.553735 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.553710    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-plugins-config-volume\") pod \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\" (UID: \"11e02dfa-4344-4948-8c7a-4931fd8d4bf7\") "
Nov 02 10:03:00.554231 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.554016    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-routing-overlay-volume" pod "11e02dfa-4344-4948-8c7a-4931fd8d4bf7" (UID: "11e02dfa-4344-4948-8c7a-4931fd8d4bf7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:03:00.554231 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.554210    4614 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-routing-overlay-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:00.554419 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.554311    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-plugins-config-volume" pod "11e02dfa-4344-4948-8c7a-4931fd8d4bf7" (UID: "11e02dfa-4344-4948-8c7a-4931fd8d4bf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:03:00.555959 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.555936    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-kube-api-access-lpfss" pod "11e02dfa-4344-4948-8c7a-4931fd8d4bf7" (UID: "11e02dfa-4344-4948-8c7a-4931fd8d4bf7"). InnerVolumeSpecName "kube-api-access-lpfss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:00.556169 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.556143    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-provider-credentials" pod "11e02dfa-4344-4948-8c7a-4931fd8d4bf7" (UID: "11e02dfa-4344-4948-8c7a-4931fd8d4bf7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:00.582859 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.582827    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm" podUID="ef7ef707-ea6c-425b-89db-ff48b54c55b9" containerName="payload-processing" containerID="cri-o://7457de31622aea41e6c6e42416f6e2ac24968b3f91465d50ebce65c8ebfd407a" gracePeriod=25
Nov 02 10:03:00.655311 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.655274    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpfss\" (UniqueName: \"kubernetes.io/projected/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-kube-api-access-lpfss\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:00.655311 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.655303    4614 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-provider-credentials\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:00.655311 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.655319    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11e02dfa-4344-4948-8c7a-4931fd8d4bf7-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:00.837982 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.837953    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:03:00.857974 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.857947    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7ef707-ea6c-425b-89db-ff48b54c55b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7ef707-ea6c-425b-89db-ff48b54c55b9-plugins-config-volume\") pod \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") "
Nov 02 10:03:00.858126 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.857998    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7ef707-ea6c-425b-89db-ff48b54c55b9-kube-api-access-h5hwn\" (UniqueName: \"kubernetes.io/projected/ef7ef707-ea6c-425b-89db-ff48b54c55b9-kube-api-access-h5hwn\") pod \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") "
Nov 02 10:03:00.858126 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.858073    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7ef707-ea6c-425b-89db-ff48b54c55b9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef7ef707-ea6c-425b-89db-ff48b54c55b9-provider-credentials\") pod \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") "
Nov 02 10:03:00.858250 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.858216    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7ef707-ea6c-425b-89db-ff48b54c55b9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef7ef707-ea6c-425b-89db-ff48b54c55b9-routing-overlay-volume\") pod \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\" (UID: \"ef7ef707-ea6c-425b-89db-ff48b54c55b9\") "
Nov 02 10:03:00.858789 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.858587    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7ef707-ea6c-425b-89db-ff48b54c55b9-plugins-config-volume" pod "ef7ef707-ea6c-425b-89db-ff48b54c55b9" (UID: "ef7ef707-ea6c-425b-89db-ff48b54c55b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:03:00.858972 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.858945    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7ef707-ea6c-425b-89db-ff48b54c55b9-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:00.859486 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.859436    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7ef707-ea6c-425b-89db-ff48b54c55b9-routing-overlay-volume" pod "ef7ef707-ea6c-425b-89db-ff48b54c55b9" (UID: "ef7ef707-ea6c-425b-89db-ff48b54c55b9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:03:00.860427 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.860399    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7ef707-ea6c-425b-89db-ff48b54c55b9-kube-api-access-h5hwn" pod "ef7ef707-ea6c-425b-89db-ff48b54c55b9" (UID: "ef7ef707-ea6c-425b-89db-ff48b54c55b9"). InnerVolumeSpecName "kube-api-access-h5hwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:00.860635 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.860620    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7ef707-ea6c-425b-89db-ff48b54c55b9-provider-credentials" pod "ef7ef707-ea6c-425b-89db-ff48b54c55b9" (UID: "ef7ef707-ea6c-425b-89db-ff48b54c55b9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:00.959644 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.959609    4614 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef7ef707-ea6c-425b-89db-ff48b54c55b9-routing-overlay-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:00.959644 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.959639    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5hwn\" (UniqueName: \"kubernetes.io/projected/ef7ef707-ea6c-425b-89db-ff48b54c55b9-kube-api-access-h5hwn\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:00.959644 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:00.959652    4614 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef7ef707-ea6c-425b-89db-ff48b54c55b9-provider-credentials\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:01.035060 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.035028    4614 generic.go:356] "Generic (PLEG): container finished" podID="11e02dfa-4344-4948-8c7a-4931fd8d4bf7" containerID="24dd152670aad1a41397a98bebbae461793357bd704c4afe99e0987667b0e721" exitCode=0
Nov 02 10:03:01.035301 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.035076    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"
Nov 02 10:03:01.035301 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.035120    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v" event={"ID":"11e02dfa-4344-4948-8c7a-4931fd8d4bf7","Type":"ContainerDied","Data":"24dd152670aad1a41397a98bebbae461793357bd704c4afe99e0987667b0e721"}
Nov 02 10:03:01.035301 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.035170    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v" event={"ID":"11e02dfa-4344-4948-8c7a-4931fd8d4bf7","Type":"ContainerDied","Data":"8bb0999729ea56330dfe77bf7b6502ab429b9bba5d449af77c9ed88c1e1f95d2"}
Nov 02 10:03:01.035301 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.035201    4614 scope.go:122] "RemoveContainer" containerID="24dd152670aad1a41397a98bebbae461793357bd704c4afe99e0987667b0e721"
Nov 02 10:03:01.036621 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.036599    4614 generic.go:356] "Generic (PLEG): container finished" podID="ef7ef707-ea6c-425b-89db-ff48b54c55b9" containerID="7457de31622aea41e6c6e42416f6e2ac24968b3f91465d50ebce65c8ebfd407a" exitCode=0
Nov 02 10:03:01.036746 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.036674    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm" event={"ID":"ef7ef707-ea6c-425b-89db-ff48b54c55b9","Type":"ContainerDied","Data":"7457de31622aea41e6c6e42416f6e2ac24968b3f91465d50ebce65c8ebfd407a"}
Nov 02 10:03:01.036746 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.036710    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm" event={"ID":"ef7ef707-ea6c-425b-89db-ff48b54c55b9","Type":"ContainerDied","Data":"d9011337567066a318ad1a1fc44ca25d20b81d14d8903f61927bdc76bc6b6ddc"}
Nov 02 10:03:01.036746 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.036724    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"
Nov 02 10:03:01.045132 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.045104    4614 scope.go:122] "RemoveContainer" containerID="24dd152670aad1a41397a98bebbae461793357bd704c4afe99e0987667b0e721"
Nov 02 10:03:01.045421 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:01.045400    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24dd152670aad1a41397a98bebbae461793357bd704c4afe99e0987667b0e721\": container with ID starting with 24dd152670aad1a41397a98bebbae461793357bd704c4afe99e0987667b0e721 not found: ID does not exist" containerID="24dd152670aad1a41397a98bebbae461793357bd704c4afe99e0987667b0e721"
Nov 02 10:03:01.045482 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.045429    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24dd152670aad1a41397a98bebbae461793357bd704c4afe99e0987667b0e721"} err="failed to get container status \"24dd152670aad1a41397a98bebbae461793357bd704c4afe99e0987667b0e721\": rpc error: code = NotFound desc = could not find container \"24dd152670aad1a41397a98bebbae461793357bd704c4afe99e0987667b0e721\": container with ID starting with 24dd152670aad1a41397a98bebbae461793357bd704c4afe99e0987667b0e721 not found: ID does not exist"
Nov 02 10:03:01.045482 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.045445    4614 scope.go:122] "RemoveContainer" containerID="7457de31622aea41e6c6e42416f6e2ac24968b3f91465d50ebce65c8ebfd407a"
Nov 02 10:03:01.054069 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.053793    4614 scope.go:122] "RemoveContainer" containerID="7457de31622aea41e6c6e42416f6e2ac24968b3f91465d50ebce65c8ebfd407a"
Nov 02 10:03:01.054143 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:01.054089    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7457de31622aea41e6c6e42416f6e2ac24968b3f91465d50ebce65c8ebfd407a\": container with ID starting with 7457de31622aea41e6c6e42416f6e2ac24968b3f91465d50ebce65c8ebfd407a not found: ID does not exist" containerID="7457de31622aea41e6c6e42416f6e2ac24968b3f91465d50ebce65c8ebfd407a"
Nov 02 10:03:01.054143 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.054125    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7457de31622aea41e6c6e42416f6e2ac24968b3f91465d50ebce65c8ebfd407a"} err="failed to get container status \"7457de31622aea41e6c6e42416f6e2ac24968b3f91465d50ebce65c8ebfd407a\": rpc error: code = NotFound desc = could not find container \"7457de31622aea41e6c6e42416f6e2ac24968b3f91465d50ebce65c8ebfd407a\": container with ID starting with 7457de31622aea41e6c6e42416f6e2ac24968b3f91465d50ebce65c8ebfd407a not found: ID does not exist"
Nov 02 10:03:01.064515 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.064490    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"]
Nov 02 10:03:01.068546 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.068493    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68n9w6v"]
Nov 02 10:03:01.083308 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.083255    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"]
Nov 02 10:03:01.086730 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.086710    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-796c85hlm"]
Nov 02 10:03:01.314890 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.314859    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e02dfa-4344-4948-8c7a-4931fd8d4bf7" path="/var/lib/kubelet/pods/11e02dfa-4344-4948-8c7a-4931fd8d4bf7/volumes"
Nov 02 10:03:01.315258 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:01.315241    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7ef707-ea6c-425b-89db-ff48b54c55b9" path="/var/lib/kubelet/pods/ef7ef707-ea6c-425b-89db-ff48b54c55b9/volumes"
Nov 02 10:03:07.866067 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:07.866034    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"]
Nov 02 10:03:07.885383 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:07.885337    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"]
Nov 02 10:03:09.634717 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:09.634677    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:03:12.866759 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:12.866688    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6" podUID="a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5" containerName="payload-processing" containerID="cri-o://d5f867723ae7c93df05299d1496120b66bd8b6996aa856ce58558b079216d5c2" gracePeriod=25
Nov 02 10:03:12.886055 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:12.886013    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7" podUID="433b0012-a699-4536-a0e0-e090a67feb6e" containerName="payload-processing" containerID="cri-o://9ffa44c2bea0756f0b86a00cb2f2ed3ad50dfc0e823350202f1ff3898fbe763b" gracePeriod=25
Nov 02 10:03:13.117274 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.117184    4614 generic.go:356] "Generic (PLEG): container finished" podID="433b0012-a699-4536-a0e0-e090a67feb6e" containerID="9ffa44c2bea0756f0b86a00cb2f2ed3ad50dfc0e823350202f1ff3898fbe763b" exitCode=0
Nov 02 10:03:13.117274 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.117215    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7" event={"ID":"433b0012-a699-4536-a0e0-e090a67feb6e","Type":"ContainerDied","Data":"9ffa44c2bea0756f0b86a00cb2f2ed3ad50dfc0e823350202f1ff3898fbe763b"}
Nov 02 10:03:13.119293 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.119262    4614 generic.go:356] "Generic (PLEG): container finished" podID="a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5" containerID="d5f867723ae7c93df05299d1496120b66bd8b6996aa856ce58558b079216d5c2" exitCode=0
Nov 02 10:03:13.119486 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.119339    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6" event={"ID":"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5","Type":"ContainerDied","Data":"d5f867723ae7c93df05299d1496120b66bd8b6996aa856ce58558b079216d5c2"}
Nov 02 10:03:13.144737 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.144719    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:03:13.172646 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.172623    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:03:13.279409 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.279378    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-kube-api-access-v9xns\" (UniqueName: \"kubernetes.io/projected/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-kube-api-access-v9xns\") pod \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") "
Nov 02 10:03:13.279409 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.279414    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-provider-credentials\") pod \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") "
Nov 02 10:03:13.279674 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.279463    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/433b0012-a699-4536-a0e0-e090a67feb6e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/433b0012-a699-4536-a0e0-e090a67feb6e-routing-overlay-volume\") pod \"433b0012-a699-4536-a0e0-e090a67feb6e\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") "
Nov 02 10:03:13.279674 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.279532    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/433b0012-a699-4536-a0e0-e090a67feb6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433b0012-a699-4536-a0e0-e090a67feb6e-plugins-config-volume\") pod \"433b0012-a699-4536-a0e0-e090a67feb6e\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") "
Nov 02 10:03:13.279674 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.279595    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433b0012-a699-4536-a0e0-e090a67feb6e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/433b0012-a699-4536-a0e0-e090a67feb6e-provider-credentials\") pod \"433b0012-a699-4536-a0e0-e090a67feb6e\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") "
Nov 02 10:03:13.279674 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.279641    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433b0012-a699-4536-a0e0-e090a67feb6e-kube-api-access-hd2bs\" (UniqueName: \"kubernetes.io/projected/433b0012-a699-4536-a0e0-e090a67feb6e-kube-api-access-hd2bs\") pod \"433b0012-a699-4536-a0e0-e090a67feb6e\" (UID: \"433b0012-a699-4536-a0e0-e090a67feb6e\") "
Nov 02 10:03:13.279904 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.279701    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-routing-overlay-volume\") pod \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") "
Nov 02 10:03:13.279904 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.279740    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-plugins-config-volume\") pod \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\" (UID: \"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5\") "
Nov 02 10:03:13.280006 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.279898    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433b0012-a699-4536-a0e0-e090a67feb6e-routing-overlay-volume" pod "433b0012-a699-4536-a0e0-e090a67feb6e" (UID: "433b0012-a699-4536-a0e0-e090a67feb6e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:03:13.280167 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.280042    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433b0012-a699-4536-a0e0-e090a67feb6e-plugins-config-volume" pod "433b0012-a699-4536-a0e0-e090a67feb6e" (UID: "433b0012-a699-4536-a0e0-e090a67feb6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:03:13.280379 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.280327    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-plugins-config-volume" pod "a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5" (UID: "a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:03:13.280379 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.280231    4614 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/433b0012-a699-4536-a0e0-e090a67feb6e-routing-overlay-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:13.280503 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.280382    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-routing-overlay-volume" pod "a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5" (UID: "a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:03:13.281698 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.281675    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-provider-credentials" pod "a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5" (UID: "a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:13.281817 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.281796    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433b0012-a699-4536-a0e0-e090a67feb6e-kube-api-access-hd2bs" pod "433b0012-a699-4536-a0e0-e090a67feb6e" (UID: "433b0012-a699-4536-a0e0-e090a67feb6e"). InnerVolumeSpecName "kube-api-access-hd2bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:13.282096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.282079    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433b0012-a699-4536-a0e0-e090a67feb6e-provider-credentials" pod "433b0012-a699-4536-a0e0-e090a67feb6e" (UID: "433b0012-a699-4536-a0e0-e090a67feb6e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:13.282140 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.282090    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-kube-api-access-v9xns" pod "a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5" (UID: "a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5"). InnerVolumeSpecName "kube-api-access-v9xns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:13.381089 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.380994    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:13.381089 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.381025    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9xns\" (UniqueName: \"kubernetes.io/projected/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-kube-api-access-v9xns\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:13.381089 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.381037    4614 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-provider-credentials\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:13.381089 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.381045    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433b0012-a699-4536-a0e0-e090a67feb6e-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:13.381089 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.381056    4614 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/433b0012-a699-4536-a0e0-e090a67feb6e-provider-credentials\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:13.381089 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.381069    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd2bs\" (UniqueName: \"kubernetes.io/projected/433b0012-a699-4536-a0e0-e090a67feb6e-kube-api-access-hd2bs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:13.381089 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:13.381083    4614 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5-routing-overlay-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:14.126580 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:14.126548    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"
Nov 02 10:03:14.127031 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:14.126541    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7" event={"ID":"433b0012-a699-4536-a0e0-e090a67feb6e","Type":"ContainerDied","Data":"5999ac2d0447e3f1fa334ea5749b7b0b8500a6cfb51da3d08a1d08c910acf443"}
Nov 02 10:03:14.127031 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:14.126683    4614 scope.go:122] "RemoveContainer" containerID="9ffa44c2bea0756f0b86a00cb2f2ed3ad50dfc0e823350202f1ff3898fbe763b"
Nov 02 10:03:14.128220 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:14.128027    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6" event={"ID":"a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5","Type":"ContainerDied","Data":"4519871d19ecd8afc4e6e97f6b049e9934c4b2aa9357844a3c4a9dc79ad651a2"}
Nov 02 10:03:14.128220 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:14.128072    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"
Nov 02 10:03:14.137294 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:14.137273    4614 scope.go:122] "RemoveContainer" containerID="d5f867723ae7c93df05299d1496120b66bd8b6996aa856ce58558b079216d5c2"
Nov 02 10:03:14.151995 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:14.151972    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"]
Nov 02 10:03:14.158264 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:14.158241    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-ddb57bhz7"]
Nov 02 10:03:14.169490 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:14.169468    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"]
Nov 02 10:03:14.174864 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:14.174840    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5qqsh6"]
Nov 02 10:03:15.340380 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:15.339689    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="433b0012-a699-4536-a0e0-e090a67feb6e" path="/var/lib/kubelet/pods/433b0012-a699-4536-a0e0-e090a67feb6e/volumes"
Nov 02 10:03:15.342884 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:15.342819    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5" path="/var/lib/kubelet/pods/a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5/volumes"
Nov 02 10:03:19.370895 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.370858    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"]
Nov 02 10:03:19.371278 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.371241    4614 cpu_manager.go:414] "containerMap: removing container" podUID="433b0012-a699-4536-a0e0-e090a67feb6e" containerName="payload-processing"
Nov 02 10:03:19.371278 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.371251    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433b0012-a699-4536-a0e0-e090a67feb6e" containerName="payload-processing"
Nov 02 10:03:19.371278 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.371269    4614 cpu_manager.go:414] "containerMap: removing container" podUID="11e02dfa-4344-4948-8c7a-4931fd8d4bf7" containerName="payload-processing"
Nov 02 10:03:19.371278 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.371274    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e02dfa-4344-4948-8c7a-4931fd8d4bf7" containerName="payload-processing"
Nov 02 10:03:19.371479 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.371286    4614 cpu_manager.go:414] "containerMap: removing container" podUID="ef7ef707-ea6c-425b-89db-ff48b54c55b9" containerName="payload-processing"
Nov 02 10:03:19.371479 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.371292    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7ef707-ea6c-425b-89db-ff48b54c55b9" containerName="payload-processing"
Nov 02 10:03:19.371479 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.371321    4614 cpu_manager.go:414] "containerMap: removing container" podUID="a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5" containerName="payload-processing"
Nov 02 10:03:19.371479 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.371327    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5" containerName="payload-processing"
Nov 02 10:03:19.371479 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.371403    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="433b0012-a699-4536-a0e0-e090a67feb6e" containerName="payload-processing"
Nov 02 10:03:19.371479 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.371412    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e02dfa-4344-4948-8c7a-4931fd8d4bf7" containerName="payload-processing"
Nov 02 10:03:19.371479 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.371420    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7ef707-ea6c-425b-89db-ff48b54c55b9" containerName="payload-processing"
Nov 02 10:03:19.371479 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.371428    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1fcffb2-7d6b-4c9d-a563-fab56f5eaaa5" containerName="payload-processing"
Nov 02 10:03:19.376066 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.376039    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.380734 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.380712    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-204fba-openshift-default-dockercfg-xvjxq\""
Nov 02 10:03:19.387563 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.387540    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"]
Nov 02 10:03:19.394645 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.394612    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"]
Nov 02 10:03:19.394767 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.394745    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.394997 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.394977    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"]
Nov 02 10:03:19.399363 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.399329    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.408782 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.408759    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"]
Nov 02 10:03:19.415488 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.415464    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"]
Nov 02 10:03:19.439341 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.439306    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.439514 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.439490    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.439592 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.439525    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.439592 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.439549    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.439592 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.439568    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.439759 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.439600    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.439759 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.439615    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pxrx\" (UniqueName: \"kubernetes.io/projected/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-kube-api-access-4pxrx\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.439759 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.439642    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.439759 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.439664    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.547119 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547079    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2222887-beb2-42c3-9404-7940c7349123-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.547337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547137    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.547337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547166    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.547337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547191    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.547337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547221    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.547337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547243    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.547337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547265    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2222887-beb2-42c3-9404-7940c7349123-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.547337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547308    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2222887-beb2-42c3-9404-7940c7349123-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.547824 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547345    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.547824 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547406    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.547824 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547466    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.547824 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547500    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.547824 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547530    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.547824 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547557    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.547824 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547582    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.547824 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547623    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.547824 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547656    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.547824 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547682    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.547824 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547732    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.547824 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547757    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pxrx\" (UniqueName: \"kubernetes.io/projected/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-kube-api-access-4pxrx\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.547824 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547796    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.548514 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547832    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.548514 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547864    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.548514 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547896    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.548514 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547929    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.548514 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.547977    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b82q5\" (UniqueName: \"kubernetes.io/projected/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-kube-api-access-b82q5\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.548514 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.548003    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2llk5\" (UniqueName: \"kubernetes.io/projected/c2222887-beb2-42c3-9404-7940c7349123-kube-api-access-2llk5\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.549435 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.549405    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.550078 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.550054    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.550205 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.550054    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.550573 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.550548    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.550714 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.550690    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.555113 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.554831    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.557602 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.557560    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.565181 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.565154    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.565760 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.565739    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pxrx\" (UniqueName: \"kubernetes.io/projected/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-kube-api-access-4pxrx\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.649469 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649394    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2222887-beb2-42c3-9404-7940c7349123-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.649469 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649448    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.649671 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649475    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.649671 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649525    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.649671 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649637    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.649839 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649675    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.649839 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649715    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.649839 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649772    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.649839 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649808    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.650038 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649848    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.650038 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649888    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b82q5\" (UniqueName: \"kubernetes.io/projected/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-kube-api-access-b82q5\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.650038 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649914    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2llk5\" (UniqueName: \"kubernetes.io/projected/c2222887-beb2-42c3-9404-7940c7349123-kube-api-access-2llk5\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.650038 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649939    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.650038 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649955    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2222887-beb2-42c3-9404-7940c7349123-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.650038 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.649977    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.650038 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.650011    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.650424 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.650041    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.650424 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.650046    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.650424 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.650071    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.650424 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.650100    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.650424 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.650124    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2222887-beb2-42c3-9404-7940c7349123-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.650424 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.650209    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.650424 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.650298    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-workload-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.650424 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.650347    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-workload-certs\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.650914 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.650474    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.650914 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.650519    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-data\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.650914 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.650582    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-credential-socket\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.651168 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.651144    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2222887-beb2-42c3-9404-7940c7349123-istiod-ca-cert\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.652378 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.652337    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2222887-beb2-42c3-9404-7940c7349123-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.652473 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.652412    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.652511 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.652463    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-podinfo\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.652627 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.652613    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-istio-envoy\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.662549 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.662507    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2222887-beb2-42c3-9404-7940c7349123-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.662827 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.662801    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b82q5\" (UniqueName: \"kubernetes.io/projected/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-kube-api-access-b82q5\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.664676 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.664658    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-token\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.665057 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.665035    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2llk5\" (UniqueName: \"kubernetes.io/projected/c2222887-beb2-42c3-9404-7940c7349123-kube-api-access-2llk5\") pod \"e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") " pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.689791 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.689766    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:19.720261 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.720234    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:19.724290 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.724211    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:19.973439 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.973402    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"]
Nov 02 10:03:19.975097 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:03:19.975069    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d382ba_ee1f_4ec7_9b70_c2193c333d3e.slice/crio-29cdf7df203e9d93bfc4a6f2fe4dde4f98b7ed0a8f923aacfe4171d2df443d51 WatchSource:0}: Error finding container 29cdf7df203e9d93bfc4a6f2fe4dde4f98b7ed0a8f923aacfe4171d2df443d51: Status 404 returned error can't find the container with id 29cdf7df203e9d93bfc4a6f2fe4dde4f98b7ed0a8f923aacfe4171d2df443d51
Nov 02 10:03:19.977443 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.977416    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:03:19.977505 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.977476    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:03:19.977540 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:19.977505    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:03:20.022930 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.022905    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"]
Nov 02 10:03:20.023913 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:03:20.023879    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2222887_beb2_42c3_9404_7940c7349123.slice/crio-e48577d6b0c3678d0b2c973437dc812afca0d1da65c7a26a4690c511c632c27a WatchSource:0}: Error finding container e48577d6b0c3678d0b2c973437dc812afca0d1da65c7a26a4690c511c632c27a: Status 404 returned error can't find the container with id e48577d6b0c3678d0b2c973437dc812afca0d1da65c7a26a4690c511c632c27a
Nov 02 10:03:20.026757 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.026729    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:03:20.026865 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.026827    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:03:20.026927 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.026876    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:03:20.032764 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.032736    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"]
Nov 02 10:03:20.036514 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:03:20.036466    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9e3aee_54bc_4bb1_8aa1_98b357aac526.slice/crio-4556c5714827efefb04d3a6071325284b2a9b26b628da25b0d49b5a3ef21a630 WatchSource:0}: Error finding container 4556c5714827efefb04d3a6071325284b2a9b26b628da25b0d49b5a3ef21a630: Status 404 returned error can't find the container with id 4556c5714827efefb04d3a6071325284b2a9b26b628da25b0d49b5a3ef21a630
Nov 02 10:03:20.039987 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.039952    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:03:20.040089 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.040023    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:03:20.040089 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.040063    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:03:20.167641 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.167606    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz" event={"ID":"c2222887-beb2-42c3-9404-7940c7349123","Type":"ContainerStarted","Data":"30581a200cacd46b21c726038a6e7d9597c0ef9bc7b3ea651db917971250d520"}
Nov 02 10:03:20.167830 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.167668    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz" event={"ID":"c2222887-beb2-42c3-9404-7940c7349123","Type":"ContainerStarted","Data":"e48577d6b0c3678d0b2c973437dc812afca0d1da65c7a26a4690c511c632c27a"}
Nov 02 10:03:20.169948 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.169917    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz" event={"ID":"1b9e3aee-54bc-4bb1-8aa1-98b357aac526","Type":"ContainerStarted","Data":"572d6d832b0557eee710e9402f08782b79d7550b7a382b747226af61fabd7de1"}
Nov 02 10:03:20.170095 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.169969    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz" event={"ID":"1b9e3aee-54bc-4bb1-8aa1-98b357aac526","Type":"ContainerStarted","Data":"4556c5714827efefb04d3a6071325284b2a9b26b628da25b0d49b5a3ef21a630"}
Nov 02 10:03:20.171818 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.171792    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr" event={"ID":"53d382ba-ee1f-4ec7-9b70-c2193c333d3e","Type":"ContainerStarted","Data":"6b0097d0a2d8bc97b7abdb33305fd2d4f723ffdd51f2d6e9e141ba3ff7c5e395"}
Nov 02 10:03:20.171927 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.171835    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr" event={"ID":"53d382ba-ee1f-4ec7-9b70-c2193c333d3e","Type":"ContainerStarted","Data":"29cdf7df203e9d93bfc4a6f2fe4dde4f98b7ed0a8f923aacfe4171d2df443d51"}
Nov 02 10:03:20.196425 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.196342    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz" podStartSLOduration=1.1963060699999999 podStartE2EDuration="1.19630607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:03:20.194143051 +0000 UTC m=+1681.026801132" watchObservedRunningTime="2026-11-02 10:03:20.19630607 +0000 UTC m=+1681.028964153"
Nov 02 10:03:20.220876 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.220815    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr" podStartSLOduration=1.220797382 podStartE2EDuration="1.220797382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:03:20.216009628 +0000 UTC m=+1681.048667701" watchObservedRunningTime="2026-11-02 10:03:20.220797382 +0000 UTC m=+1681.053455463"
Nov 02 10:03:20.243260 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.243173    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz" podStartSLOduration=1.2431570220000001 podStartE2EDuration="1.243157022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:03:20.240167033 +0000 UTC m=+1681.072825115" watchObservedRunningTime="2026-11-02 10:03:20.243157022 +0000 UTC m=+1681.075815095"
Nov 02 10:03:20.700291 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.700191    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:20.720454 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.720418    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:20.724648 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:20.724619    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:21.693378 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:21.693213    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 10:03:21.693378 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:21.693280    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr" podUID="53d382ba-ee1f-4ec7-9b70-c2193c333d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 10:03:21.722368 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:21.722309    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 10:03:21.722788 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:21.722386    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz" podUID="c2222887-beb2-42c3-9404-7940c7349123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 10:03:21.771655 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:21.729742    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 10:03:21.771655 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:21.729795    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz" podUID="1b9e3aee-54bc-4bb1-8aa1-98b357aac526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 10:03:22.697041 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:22.696298    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 10:03:22.697041 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:22.696385    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr" podUID="53d382ba-ee1f-4ec7-9b70-c2193c333d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 10:03:22.720771 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:22.720733    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 10:03:22.720945 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:22.720780    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz" podUID="c2222887-beb2-42c3-9404-7940c7349123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 10:03:22.725319 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:22.725301    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 10:03:22.725743 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:22.725333    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz" podUID="1b9e3aee-54bc-4bb1-8aa1-98b357aac526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 10:03:23.695379 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:23.692761    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 10:03:23.695379 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:23.692823    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr" podUID="53d382ba-ee1f-4ec7-9b70-c2193c333d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 10:03:23.720696 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:23.720648    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 10:03:23.720832 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:23.720711    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz" podUID="c2222887-beb2-42c3-9404-7940c7349123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 10:03:23.724976 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:23.724958    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 10:03:23.725078 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:23.724992    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz" podUID="1b9e3aee-54bc-4bb1-8aa1-98b357aac526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 10:03:24.696384 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:24.694769    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 10:03:24.696384 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:24.694839    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr" podUID="53d382ba-ee1f-4ec7-9b70-c2193c333d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 10:03:24.721735 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:24.721455    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 10:03:24.721735 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:24.721495    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz" podUID="c2222887-beb2-42c3-9404-7940c7349123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 10:03:24.725348 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:24.725280    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 10:03:24.725348 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:24.725312    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz" podUID="1b9e3aee-54bc-4bb1-8aa1-98b357aac526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 10:03:25.694375 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:25.693780    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 10:03:25.694375 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:25.693848    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr" podUID="53d382ba-ee1f-4ec7-9b70-c2193c333d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 10:03:25.720813 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:25.720769    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 02 10:03:25.721332 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:25.720823    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz" podUID="c2222887-beb2-42c3-9404-7940c7349123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 02 10:03:25.725128 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:25.725105    4614 patch_prober.go:29] interesting pod/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 02 10:03:25.725175 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:25.725146    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz" podUID="1b9e3aee-54bc-4bb1-8aa1-98b357aac526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 02 10:03:26.841957 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:26.841924    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:26.844343 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:26.844323    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:26.850281 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:26.850261    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:03:26.928600 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:26.928573    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:26.938808 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:26.938778    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:26.939480 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:26.939464    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:03:26.955749 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:26.955720    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:26.956441 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:26.956416    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:26.957431 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:26.957409    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:03:39.635830 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:39.635786    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:03:40.824375 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.822315    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"]
Nov 02 10:03:40.833726 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.833683    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:40.868877 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.854753    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"]
Nov 02 10:03:40.868877 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.866150    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-e2e--e7669bbb-dockercfg-s8xkv\""
Nov 02 10:03:40.868877 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.866222    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"kube-root-ca.crt\""
Nov 02 10:03:40.868877 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.866290    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"routing-overlay\""
Nov 02 10:03:40.868877 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.866371    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"payload-processing-external-model-plugins-e2e-extmod-b-204fba\""
Nov 02 10:03:40.868877 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.866421    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-407717c0a9b4\""
Nov 02 10:03:40.868877 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.866469    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"openshift-service-ca.crt\""
Nov 02 10:03:40.868877 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.866511    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-d1ad6b3bd8a0\""
Nov 02 10:03:40.868877 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.866570    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-204fba\"/\"agc-cred-f77d6b5159d1\""
Nov 02 10:03:40.963400 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.963336    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:40.963576 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.963449    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxzpl\" (UniqueName: \"kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-kube-api-access-cxzpl\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:40.963576 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.963514    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:40.963660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:40.963590    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:41.023088 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.023055    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"]
Nov 02 10:03:41.023254 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.023230    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn" podUID="91a9290d-00fc-4af7-a072-60c2206112c3" containerName="sim" containerID="cri-o://840f4fd5a401d35036e92d2053810ed68ec93f4a00c869e665bf83bd04d85133" gracePeriod=30
Nov 02 10:03:41.065008 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.064976    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:41.065132 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.065119    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:41.065181 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.065160    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxzpl\" (UniqueName: \"kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-kube-api-access-cxzpl\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:41.065228 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.065212    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:41.065790 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.065765    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:41.065972 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.065952    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:41.067678 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.067659    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:41.077541 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.077492    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxzpl\" (UniqueName: \"kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-kube-api-access-cxzpl\") pod \"payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") " pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:41.175864 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.175817    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:41.365818 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.365735    4614 generic.go:356] "Generic (PLEG): container finished" podID="91a9290d-00fc-4af7-a072-60c2206112c3" containerID="840f4fd5a401d35036e92d2053810ed68ec93f4a00c869e665bf83bd04d85133" exitCode=0
Nov 02 10:03:41.365964 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.365811    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn" event={"ID":"91a9290d-00fc-4af7-a072-60c2206112c3","Type":"ContainerDied","Data":"840f4fd5a401d35036e92d2053810ed68ec93f4a00c869e665bf83bd04d85133"}
Nov 02 10:03:41.450530 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.450500    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"
Nov 02 10:03:41.471366 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.471322    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a9290d-00fc-4af7-a072-60c2206112c3-kube-api-access-tq756\" (UniqueName: \"kubernetes.io/projected/91a9290d-00fc-4af7-a072-60c2206112c3-kube-api-access-tq756\") pod \"91a9290d-00fc-4af7-a072-60c2206112c3\" (UID: \"91a9290d-00fc-4af7-a072-60c2206112c3\") "
Nov 02 10:03:41.471517 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.471406    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91a9290d-00fc-4af7-a072-60c2206112c3-tls\" (UniqueName: \"kubernetes.io/secret/91a9290d-00fc-4af7-a072-60c2206112c3-tls\") pod \"91a9290d-00fc-4af7-a072-60c2206112c3\" (UID: \"91a9290d-00fc-4af7-a072-60c2206112c3\") "
Nov 02 10:03:41.473546 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.473506    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a9290d-00fc-4af7-a072-60c2206112c3-tls" pod "91a9290d-00fc-4af7-a072-60c2206112c3" (UID: "91a9290d-00fc-4af7-a072-60c2206112c3"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:03:41.473741 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.473702    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a9290d-00fc-4af7-a072-60c2206112c3-kube-api-access-tq756" pod "91a9290d-00fc-4af7-a072-60c2206112c3" (UID: "91a9290d-00fc-4af7-a072-60c2206112c3"). InnerVolumeSpecName "kube-api-access-tq756". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:41.498571 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.498546    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"]
Nov 02 10:03:41.501303 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:03:41.501273    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ce62a8_8fd1_400f_8fb6_c81574f5ed65.slice/crio-e58eca97a9a0169986df0f013090258a804353f9844b87d7d34a5400efb1993a WatchSource:0}: Error finding container e58eca97a9a0169986df0f013090258a804353f9844b87d7d34a5400efb1993a: Status 404 returned error can't find the container with id e58eca97a9a0169986df0f013090258a804353f9844b87d7d34a5400efb1993a
Nov 02 10:03:41.572650 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.572624    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq756\" (UniqueName: \"kubernetes.io/projected/91a9290d-00fc-4af7-a072-60c2206112c3-kube-api-access-tq756\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:41.572650 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:41.572649    4614 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/91a9290d-00fc-4af7-a072-60c2206112c3-tls\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:42.372117 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.372029    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77" event={"ID":"99ce62a8-8fd1-400f-8fb6-c81574f5ed65","Type":"ContainerStarted","Data":"fe6b01dd7a5604b09d4cc1bd12e1ed57b6ab2f09a48426297f2e9787cd21fb4e"}
Nov 02 10:03:42.372117 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.372092    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:42.372608 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.372116    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77" event={"ID":"99ce62a8-8fd1-400f-8fb6-c81574f5ed65","Type":"ContainerStarted","Data":"e58eca97a9a0169986df0f013090258a804353f9844b87d7d34a5400efb1993a"}
Nov 02 10:03:42.373501 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.373467    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"
Nov 02 10:03:42.373501 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.373494    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn" event={"ID":"91a9290d-00fc-4af7-a072-60c2206112c3","Type":"ContainerDied","Data":"e239d99bede6cf114cc82ddc63d47aa2c278a3a82aa3f77bc25cc0e5c441f026"}
Nov 02 10:03:42.373677 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.373530    4614 scope.go:122] "RemoveContainer" containerID="840f4fd5a401d35036e92d2053810ed68ec93f4a00c869e665bf83bd04d85133"
Nov 02 10:03:42.393930 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.393888    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77" podStartSLOduration=2.393873793 podStartE2EDuration="2.393873793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:03:42.392220626 +0000 UTC m=+1703.224878706" watchObservedRunningTime="2026-11-02 10:03:42.393873793 +0000 UTC m=+1703.226531873"
Nov 02 10:03:42.406594 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.406570    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"]
Nov 02 10:03:42.410636 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.410615    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-c2f625/e2e-usage-sim-78b35d-59c776c7c7-gbsrn"]
Nov 02 10:03:42.983866 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.983827    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"]
Nov 02 10:03:42.984348 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.984332    4614 cpu_manager.go:414] "containerMap: removing container" podUID="91a9290d-00fc-4af7-a072-60c2206112c3" containerName="sim"
Nov 02 10:03:42.984348 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.984372    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a9290d-00fc-4af7-a072-60c2206112c3" containerName="sim"
Nov 02 10:03:42.984513 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.984445    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a9290d-00fc-4af7-a072-60c2206112c3" containerName="sim"
Nov 02 10:03:42.989049 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.989027    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:42.989729 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.989697    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ceee9859-b557-4daa-8027-bf45e3d59e82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:42.989859 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.989751    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ceee9859-b557-4daa-8027-bf45e3d59e82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:42.989930 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.989864    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceee9859-b557-4daa-8027-bf45e3d59e82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:42.989930 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.989915    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8lg\" (UniqueName: \"kubernetes.io/projected/ceee9859-b557-4daa-8027-bf45e3d59e82-kube-api-access-lt8lg\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:42.997219 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.997194    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-plugins-e2e-extmod-a-629129\""
Nov 02 10:03:42.997438 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.997418    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"openshift-service-ca.crt\""
Nov 02 10:03:42.997550 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.997534    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-6ce0a95c3878\""
Nov 02 10:03:42.997615 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.997571    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"agc-cred-8eb9fb76a028\""
Nov 02 10:03:42.997731 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.997703    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"kube-root-ca.crt\""
Nov 02 10:03:42.998514 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.998492    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"payload-processing-external-model-e2e--7be3f336-dockercfg-zbbxs\""
Nov 02 10:03:42.998935 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:42.998907    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-629129\"/\"routing-overlay\""
Nov 02 10:03:43.002546 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:43.002524    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"]
Nov 02 10:03:43.090532 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:43.090498    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ceee9859-b557-4daa-8027-bf45e3d59e82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:43.090733 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:43.090565    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceee9859-b557-4daa-8027-bf45e3d59e82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:43.090733 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:43.090605    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8lg\" (UniqueName: \"kubernetes.io/projected/ceee9859-b557-4daa-8027-bf45e3d59e82-kube-api-access-lt8lg\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:43.090733 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:43.090710    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ceee9859-b557-4daa-8027-bf45e3d59e82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:43.091408 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:43.091345    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ceee9859-b557-4daa-8027-bf45e3d59e82-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:43.091554 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:43.091521    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceee9859-b557-4daa-8027-bf45e3d59e82-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:43.093250 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:43.093222    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ceee9859-b557-4daa-8027-bf45e3d59e82-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:43.103478 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:43.103448    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8lg\" (UniqueName: \"kubernetes.io/projected/ceee9859-b557-4daa-8027-bf45e3d59e82-kube-api-access-lt8lg\") pod \"payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") " pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:43.268435 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:43.267772    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"]
Nov 02 10:03:43.307646 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:43.305647    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:43.325450 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:43.325259    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a9290d-00fc-4af7-a072-60c2206112c3" path="/var/lib/kubelet/pods/91a9290d-00fc-4af7-a072-60c2206112c3/volumes"
Nov 02 10:03:43.497867 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:43.497827    4614 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-407717c0a9b4: secret "agc-cred-407717c0a9b4" not found
Nov 02 10:03:43.497867 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:43.497859    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77: secret "agc-cred-407717c0a9b4" not found
Nov 02 10:03:43.498392 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:43.497923    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials podName:99ce62a8-8fd1-400f-8fb6-c81574f5ed65 nodeName:}" failed. No retries permitted until 2026-11-02 10:03:43.997903744 +0000 UTC m=+1704.830561816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77" (UID: "99ce62a8-8fd1-400f-8fb6-c81574f5ed65") : secret "agc-cred-407717c0a9b4" not found
Nov 02 10:03:43.600736 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:43.600713    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"]
Nov 02 10:03:43.602420 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:03:43.602383    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceee9859_b557_4daa_8027_bf45e3d59e82.slice/crio-0c012e3b941ad3aadefeb235605a1c90c027eb93d305e5d69e336487a2d16090 WatchSource:0}: Error finding container 0c012e3b941ad3aadefeb235605a1c90c027eb93d305e5d69e336487a2d16090: Status 404 returned error can't find the container with id 0c012e3b941ad3aadefeb235605a1c90c027eb93d305e5d69e336487a2d16090
Nov 02 10:03:44.005592 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:44.005552    4614 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-407717c0a9b4: secret "agc-cred-407717c0a9b4" not found
Nov 02 10:03:44.005592 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:44.005587    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77: secret "agc-cred-407717c0a9b4" not found
Nov 02 10:03:44.005784 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:44.005640    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials podName:99ce62a8-8fd1-400f-8fb6-c81574f5ed65 nodeName:}" failed. No retries permitted until 2026-11-02 10:03:45.005625415 +0000 UTC m=+1705.838283473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77" (UID: "99ce62a8-8fd1-400f-8fb6-c81574f5ed65") : secret "agc-cred-407717c0a9b4" not found
Nov 02 10:03:44.412186 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:44.412088    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r" event={"ID":"ceee9859-b557-4daa-8027-bf45e3d59e82","Type":"ContainerStarted","Data":"8be650c50eebd6efba088bf5700ea14966044cb8285a4a8b82d64e3a95bc4489"}
Nov 02 10:03:44.412389 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:44.412291    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:03:44.412389 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:44.412314    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r" event={"ID":"ceee9859-b557-4daa-8027-bf45e3d59e82","Type":"ContainerStarted","Data":"0c012e3b941ad3aadefeb235605a1c90c027eb93d305e5d69e336487a2d16090"}
Nov 02 10:03:44.433819 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:44.433775    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r" podStartSLOduration=2.433762293 podStartE2EDuration="2.433762293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:03:44.431710189 +0000 UTC m=+1705.264368269" watchObservedRunningTime="2026-11-02 10:03:44.433762293 +0000 UTC m=+1705.266420373"
Nov 02 10:03:45.031378 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:45.026878    4614 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-407717c0a9b4: secret "agc-cred-407717c0a9b4" not found
Nov 02 10:03:45.031378 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:45.026925    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77: secret "agc-cred-407717c0a9b4" not found
Nov 02 10:03:45.031378 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:45.026988    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials podName:99ce62a8-8fd1-400f-8fb6-c81574f5ed65 nodeName:}" failed. No retries permitted until 2026-11-02 10:03:47.026968853 +0000 UTC m=+1707.859626912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77" (UID: "99ce62a8-8fd1-400f-8fb6-c81574f5ed65") : secret "agc-cred-407717c0a9b4" not found
Nov 02 10:03:47.046507 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:47.046475    4614 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-204fba/agc-cred-407717c0a9b4: secret "agc-cred-407717c0a9b4" not found
Nov 02 10:03:47.046507 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:47.046508    4614 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77: secret "agc-cred-407717c0a9b4" not found
Nov 02 10:03:47.046940 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:47.046561    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials podName:99ce62a8-8fd1-400f-8fb6-c81574f5ed65 nodeName:}" failed. No retries permitted until 2026-11-02 10:03:51.046547229 +0000 UTC m=+1711.879205287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77" (UID: "99ce62a8-8fd1-400f-8fb6-c81574f5ed65") : secret "agc-cred-407717c0a9b4" not found
Nov 02 10:03:49.412538 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:49.412481    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77" podUID="99ce62a8-8fd1-400f-8fb6-c81574f5ed65" containerName="payload-processing" containerID="cri-o://fe6b01dd7a5604b09d4cc1bd12e1ed57b6ab2f09a48426297f2e9787cd21fb4e" gracePeriod=25
Nov 02 10:03:49.670607 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:49.670544    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:49.776240 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:49.776206    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-routing-overlay-volume\") pod \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") "
Nov 02 10:03:49.776429 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:49.776283    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-plugins-config-volume\") pod \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") "
Nov 02 10:03:49.776429 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:49.776410    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials\") pod \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") "
Nov 02 10:03:49.776574 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:49.776461    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-kube-api-access-cxzpl\" (UniqueName: \"kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-kube-api-access-cxzpl\") pod \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\" (UID: \"99ce62a8-8fd1-400f-8fb6-c81574f5ed65\") "
Nov 02 10:03:49.776636 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:49.776616    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-routing-overlay-volume" pod "99ce62a8-8fd1-400f-8fb6-c81574f5ed65" (UID: "99ce62a8-8fd1-400f-8fb6-c81574f5ed65"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:03:49.776707 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:49.776684    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-plugins-config-volume" pod "99ce62a8-8fd1-400f-8fb6-c81574f5ed65" (UID: "99ce62a8-8fd1-400f-8fb6-c81574f5ed65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:03:49.776891 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:49.776876    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:49.776967 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:49.776892    4614 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-routing-overlay-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:49.778534 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:49.778512    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-kube-api-access-cxzpl" pod "99ce62a8-8fd1-400f-8fb6-c81574f5ed65" (UID: "99ce62a8-8fd1-400f-8fb6-c81574f5ed65"). InnerVolumeSpecName "kube-api-access-cxzpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:49.778628 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:49.778602    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials" pod "99ce62a8-8fd1-400f-8fb6-c81574f5ed65" (UID: "99ce62a8-8fd1-400f-8fb6-c81574f5ed65"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:03:49.878196 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:49.878166    4614 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-provider-credentials\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:49.878196 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:49.878192    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxzpl\" (UniqueName: \"kubernetes.io/projected/99ce62a8-8fd1-400f-8fb6-c81574f5ed65-kube-api-access-cxzpl\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:03:50.443232 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:50.443195    4614 generic.go:356] "Generic (PLEG): container finished" podID="99ce62a8-8fd1-400f-8fb6-c81574f5ed65" containerID="fe6b01dd7a5604b09d4cc1bd12e1ed57b6ab2f09a48426297f2e9787cd21fb4e" exitCode=0
Nov 02 10:03:50.443734 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:50.443258    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"
Nov 02 10:03:50.443734 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:50.443281    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77" event={"ID":"99ce62a8-8fd1-400f-8fb6-c81574f5ed65","Type":"ContainerDied","Data":"fe6b01dd7a5604b09d4cc1bd12e1ed57b6ab2f09a48426297f2e9787cd21fb4e"}
Nov 02 10:03:50.443734 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:50.443334    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77" event={"ID":"99ce62a8-8fd1-400f-8fb6-c81574f5ed65","Type":"ContainerDied","Data":"e58eca97a9a0169986df0f013090258a804353f9844b87d7d34a5400efb1993a"}
Nov 02 10:03:50.443734 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:50.443373    4614 scope.go:122] "RemoveContainer" containerID="fe6b01dd7a5604b09d4cc1bd12e1ed57b6ab2f09a48426297f2e9787cd21fb4e"
Nov 02 10:03:50.453068 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:50.453053    4614 scope.go:122] "RemoveContainer" containerID="fe6b01dd7a5604b09d4cc1bd12e1ed57b6ab2f09a48426297f2e9787cd21fb4e"
Nov 02 10:03:50.453289 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:03:50.453272    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6b01dd7a5604b09d4cc1bd12e1ed57b6ab2f09a48426297f2e9787cd21fb4e\": container with ID starting with fe6b01dd7a5604b09d4cc1bd12e1ed57b6ab2f09a48426297f2e9787cd21fb4e not found: ID does not exist" containerID="fe6b01dd7a5604b09d4cc1bd12e1ed57b6ab2f09a48426297f2e9787cd21fb4e"
Nov 02 10:03:50.453329 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:50.453297    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6b01dd7a5604b09d4cc1bd12e1ed57b6ab2f09a48426297f2e9787cd21fb4e"} err="failed to get container status \"fe6b01dd7a5604b09d4cc1bd12e1ed57b6ab2f09a48426297f2e9787cd21fb4e\": rpc error: code = NotFound desc = could not find container \"fe6b01dd7a5604b09d4cc1bd12e1ed57b6ab2f09a48426297f2e9787cd21fb4e\": container with ID starting with fe6b01dd7a5604b09d4cc1bd12e1ed57b6ab2f09a48426297f2e9787cd21fb4e not found: ID does not exist"
Nov 02 10:03:50.467459 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:50.467437    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"]
Nov 02 10:03:50.471930 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:50.471910    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-204fba/payload-processing-external-model-e2e-extmod-b-204fba-8c68cxg77"]
Nov 02 10:03:51.314691 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:51.314651    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ce62a8-8fd1-400f-8fb6-c81574f5ed65" path="/var/lib/kubelet/pods/99ce62a8-8fd1-400f-8fb6-c81574f5ed65/volumes"
Nov 02 10:03:55.420092 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:03:55.420062    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:04:09.636059 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:04:09.636025    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:04:17.440440 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:17.440408    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"]
Nov 02 10:04:22.441252 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.441204    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r" podUID="ceee9859-b557-4daa-8027-bf45e3d59e82" containerName="payload-processing" containerID="cri-o://8be650c50eebd6efba088bf5700ea14966044cb8285a4a8b82d64e3a95bc4489" gracePeriod=25
Nov 02 10:04:22.655311 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.655277    4614 generic.go:356] "Generic (PLEG): container finished" podID="ceee9859-b557-4daa-8027-bf45e3d59e82" containerID="8be650c50eebd6efba088bf5700ea14966044cb8285a4a8b82d64e3a95bc4489" exitCode=0
Nov 02 10:04:22.655478 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.655334    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r" event={"ID":"ceee9859-b557-4daa-8027-bf45e3d59e82","Type":"ContainerDied","Data":"8be650c50eebd6efba088bf5700ea14966044cb8285a4a8b82d64e3a95bc4489"}
Nov 02 10:04:22.693277 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.693217    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:04:22.830086 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.830055    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceee9859-b557-4daa-8027-bf45e3d59e82-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ceee9859-b557-4daa-8027-bf45e3d59e82-provider-credentials\") pod \"ceee9859-b557-4daa-8027-bf45e3d59e82\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") "
Nov 02 10:04:22.830264 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.830146    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceee9859-b557-4daa-8027-bf45e3d59e82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceee9859-b557-4daa-8027-bf45e3d59e82-plugins-config-volume\") pod \"ceee9859-b557-4daa-8027-bf45e3d59e82\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") "
Nov 02 10:04:22.830264 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.830221    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceee9859-b557-4daa-8027-bf45e3d59e82-kube-api-access-lt8lg\" (UniqueName: \"kubernetes.io/projected/ceee9859-b557-4daa-8027-bf45e3d59e82-kube-api-access-lt8lg\") pod \"ceee9859-b557-4daa-8027-bf45e3d59e82\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") "
Nov 02 10:04:22.830264 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.830248    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceee9859-b557-4daa-8027-bf45e3d59e82-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ceee9859-b557-4daa-8027-bf45e3d59e82-routing-overlay-volume\") pod \"ceee9859-b557-4daa-8027-bf45e3d59e82\" (UID: \"ceee9859-b557-4daa-8027-bf45e3d59e82\") "
Nov 02 10:04:22.830642 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.830616    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceee9859-b557-4daa-8027-bf45e3d59e82-routing-overlay-volume" pod "ceee9859-b557-4daa-8027-bf45e3d59e82" (UID: "ceee9859-b557-4daa-8027-bf45e3d59e82"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:04:22.830642 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.830629    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceee9859-b557-4daa-8027-bf45e3d59e82-plugins-config-volume" pod "ceee9859-b557-4daa-8027-bf45e3d59e82" (UID: "ceee9859-b557-4daa-8027-bf45e3d59e82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:04:22.832268 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.832239    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceee9859-b557-4daa-8027-bf45e3d59e82-provider-credentials" pod "ceee9859-b557-4daa-8027-bf45e3d59e82" (UID: "ceee9859-b557-4daa-8027-bf45e3d59e82"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:04:22.832384 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.832338    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceee9859-b557-4daa-8027-bf45e3d59e82-kube-api-access-lt8lg" pod "ceee9859-b557-4daa-8027-bf45e3d59e82" (UID: "ceee9859-b557-4daa-8027-bf45e3d59e82"). InnerVolumeSpecName "kube-api-access-lt8lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:04:22.935325 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.933417    4614 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ceee9859-b557-4daa-8027-bf45e3d59e82-provider-credentials\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:22.935325 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.933457    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceee9859-b557-4daa-8027-bf45e3d59e82-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:22.935325 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.933473    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8lg\" (UniqueName: \"kubernetes.io/projected/ceee9859-b557-4daa-8027-bf45e3d59e82-kube-api-access-lt8lg\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:22.935325 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:22.933488    4614 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ceee9859-b557-4daa-8027-bf45e3d59e82-routing-overlay-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:23.661999 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:23.661920    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r" event={"ID":"ceee9859-b557-4daa-8027-bf45e3d59e82","Type":"ContainerDied","Data":"0c012e3b941ad3aadefeb235605a1c90c027eb93d305e5d69e336487a2d16090"}
Nov 02 10:04:23.661999 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:23.661960    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"
Nov 02 10:04:23.661999 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:23.661982    4614 scope.go:122] "RemoveContainer" containerID="8be650c50eebd6efba088bf5700ea14966044cb8285a4a8b82d64e3a95bc4489"
Nov 02 10:04:23.679820 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:23.679795    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"]
Nov 02 10:04:23.683756 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:23.683736    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-629129/payload-processing-external-model-e2e-extmod-a-629129-76d5m2r5r"]
Nov 02 10:04:25.314625 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:25.314589    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceee9859-b557-4daa-8027-bf45e3d59e82" path="/var/lib/kubelet/pods/ceee9859-b557-4daa-8027-bf45e3d59e82/volumes"
Nov 02 10:04:39.635762 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:04:39.635723    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:04:43.835497 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:43.835458    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"]
Nov 02 10:04:43.836096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:43.835715    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm" podUID="7e3ea645-ba8e-4ebb-8332-1617c13436be" containerName="maas-api" containerID="cri-o://b9ef58d191fc7d2b4074388b731b5d97995e784c318c368ac429cbe4d694554c" gracePeriod=30
Nov 02 10:04:44.177267 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.177235    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 10:04:44.263068 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.263036    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-maas-api-tls\") pod \"7e3ea645-ba8e-4ebb-8332-1617c13436be\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") "
Nov 02 10:04:44.263265 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.263091    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3ea645-ba8e-4ebb-8332-1617c13436be-kube-api-access-cxxwk\" (UniqueName: \"kubernetes.io/projected/7e3ea645-ba8e-4ebb-8332-1617c13436be-kube-api-access-cxxwk\") pod \"7e3ea645-ba8e-4ebb-8332-1617c13436be\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") "
Nov 02 10:04:44.263265 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.263185    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-metrics-certs\") pod \"7e3ea645-ba8e-4ebb-8332-1617c13436be\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") "
Nov 02 10:04:44.263265 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.263229    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e3ea645-ba8e-4ebb-8332-1617c13436be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3ea645-ba8e-4ebb-8332-1617c13436be-openshift-service-ca\") pod \"7e3ea645-ba8e-4ebb-8332-1617c13436be\" (UID: \"7e3ea645-ba8e-4ebb-8332-1617c13436be\") "
Nov 02 10:04:44.263727 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.263701    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3ea645-ba8e-4ebb-8332-1617c13436be-openshift-service-ca" pod "7e3ea645-ba8e-4ebb-8332-1617c13436be" (UID: "7e3ea645-ba8e-4ebb-8332-1617c13436be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:04:44.265254 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.265231    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-maas-api-tls" pod "7e3ea645-ba8e-4ebb-8332-1617c13436be" (UID: "7e3ea645-ba8e-4ebb-8332-1617c13436be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:04:44.265340 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.265261    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-metrics-certs" pod "7e3ea645-ba8e-4ebb-8332-1617c13436be" (UID: "7e3ea645-ba8e-4ebb-8332-1617c13436be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:04:44.265340 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.265306    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3ea645-ba8e-4ebb-8332-1617c13436be-kube-api-access-cxxwk" pod "7e3ea645-ba8e-4ebb-8332-1617c13436be" (UID: "7e3ea645-ba8e-4ebb-8332-1617c13436be"). InnerVolumeSpecName "kube-api-access-cxxwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:04:44.365280 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.365229    4614 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-metrics-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:44.365280 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.365280    4614 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3ea645-ba8e-4ebb-8332-1617c13436be-openshift-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:44.365575 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.365298    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3ea645-ba8e-4ebb-8332-1617c13436be-maas-api-tls\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:44.365575 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.365312    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxxwk\" (UniqueName: \"kubernetes.io/projected/7e3ea645-ba8e-4ebb-8332-1617c13436be-kube-api-access-cxxwk\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:44.782678 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.782642    4614 generic.go:356] "Generic (PLEG): container finished" podID="7e3ea645-ba8e-4ebb-8332-1617c13436be" containerID="b9ef58d191fc7d2b4074388b731b5d97995e784c318c368ac429cbe4d694554c" exitCode=0
Nov 02 10:04:44.782885 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.782692    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"
Nov 02 10:04:44.782885 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.782723    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm" event={"ID":"7e3ea645-ba8e-4ebb-8332-1617c13436be","Type":"ContainerDied","Data":"b9ef58d191fc7d2b4074388b731b5d97995e784c318c368ac429cbe4d694554c"}
Nov 02 10:04:44.782885 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.782774    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm" event={"ID":"7e3ea645-ba8e-4ebb-8332-1617c13436be","Type":"ContainerDied","Data":"bb0ecced87a1404f5becc5acb6c6710f1e8265e72fd1f40550cd11c64ad26fa3"}
Nov 02 10:04:44.782885 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.782797    4614 scope.go:122] "RemoveContainer" containerID="b9ef58d191fc7d2b4074388b731b5d97995e784c318c368ac429cbe4d694554c"
Nov 02 10:04:44.792477 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.792459    4614 scope.go:122] "RemoveContainer" containerID="b9ef58d191fc7d2b4074388b731b5d97995e784c318c368ac429cbe4d694554c"
Nov 02 10:04:44.792718 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:04:44.792699    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ef58d191fc7d2b4074388b731b5d97995e784c318c368ac429cbe4d694554c\": container with ID starting with b9ef58d191fc7d2b4074388b731b5d97995e784c318c368ac429cbe4d694554c not found: ID does not exist" containerID="b9ef58d191fc7d2b4074388b731b5d97995e784c318c368ac429cbe4d694554c"
Nov 02 10:04:44.792767 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.792725    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ef58d191fc7d2b4074388b731b5d97995e784c318c368ac429cbe4d694554c"} err="failed to get container status \"b9ef58d191fc7d2b4074388b731b5d97995e784c318c368ac429cbe4d694554c\": rpc error: code = NotFound desc = could not find container \"b9ef58d191fc7d2b4074388b731b5d97995e784c318c368ac429cbe4d694554c\": container with ID starting with b9ef58d191fc7d2b4074388b731b5d97995e784c318c368ac429cbe4d694554c not found: ID does not exist"
Nov 02 10:04:44.805993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.805971    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"]
Nov 02 10:04:44.810249 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:44.810227    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-629129-84d76fcfbf-4rdlm"]
Nov 02 10:04:45.316453 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:45.316421    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3ea645-ba8e-4ebb-8332-1617c13436be" path="/var/lib/kubelet/pods/7e3ea645-ba8e-4ebb-8332-1617c13436be/volumes"
Nov 02 10:04:50.911925 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:50.911891    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"]
Nov 02 10:04:50.912370 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:50.912136    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn" podUID="4c42817a-b1ca-486d-8f46-3db3bbf9346c" containerName="istio-proxy" containerID="cri-o://06f7f1fa25c65eead796e403001245f5692e34630823f7ed59a085f14bc3ec0a" gracePeriod=30
Nov 02 10:04:50.917044 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:50.917021    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"]
Nov 02 10:04:50.917230 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:50.917207    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx" podUID="2e4b65a7-462d-4831-8cca-65f29dac08df" containerName="istio-proxy" containerID="cri-o://d11e1e11e374f42839eaa4e03c5b96e2a6e3d5ec2505793c0e6e81112b9177ac" gracePeriod=30
Nov 02 10:04:55.997168 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:55.997140    4614 patch_prober.go:29] interesting pod/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 10:04:55.997525 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:55.997191    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx" podUID="2e4b65a7-462d-4831-8cca-65f29dac08df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 10:04:56.182829 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.182760    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 10:04:56.229818 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.229782    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:04:56.302541 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302509    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c42817a-b1ca-486d-8f46-3db3bbf9346c-kube-api-access-74kc2\" (UniqueName: \"kubernetes.io/projected/4c42817a-b1ca-486d-8f46-3db3bbf9346c-kube-api-access-74kc2\") pod \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") "
Nov 02 10:04:56.302720 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302549    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-data\") pod \"2e4b65a7-462d-4831-8cca-65f29dac08df\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") "
Nov 02 10:04:56.302720 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302598    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-credential-socket\") pod \"2e4b65a7-462d-4831-8cca-65f29dac08df\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") "
Nov 02 10:04:56.302720 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302628    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4b65a7-462d-4831-8cca-65f29dac08df-kube-api-access-699xx\" (UniqueName: \"kubernetes.io/projected/2e4b65a7-462d-4831-8cca-65f29dac08df-kube-api-access-699xx\") pod \"2e4b65a7-462d-4831-8cca-65f29dac08df\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") "
Nov 02 10:04:56.302720 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302659    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-workload-certs\") pod \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") "
Nov 02 10:04:56.302720 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302719    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-podinfo\") pod \"2e4b65a7-462d-4831-8cca-65f29dac08df\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") "
Nov 02 10:04:56.303003 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302755    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-token\" (UniqueName: \"kubernetes.io/projected/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-token\") pod \"2e4b65a7-462d-4831-8cca-65f29dac08df\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") "
Nov 02 10:04:56.303003 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302783    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-workload-socket\") pod \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") "
Nov 02 10:04:56.303003 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302816    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-credential-socket\") pod \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") "
Nov 02 10:04:56.303003 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302869    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4b65a7-462d-4831-8cca-65f29dac08df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4b65a7-462d-4831-8cca-65f29dac08df-istiod-ca-cert\") pod \"2e4b65a7-462d-4831-8cca-65f29dac08df\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") "
Nov 02 10:04:56.303003 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302898    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-envoy\") pod \"2e4b65a7-462d-4831-8cca-65f29dac08df\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") "
Nov 02 10:04:56.303003 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302905    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-data" pod "2e4b65a7-462d-4831-8cca-65f29dac08df" (UID: "2e4b65a7-462d-4831-8cca-65f29dac08df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.303003 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302934    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-data\") pod \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") "
Nov 02 10:04:56.303003 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302966    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-workload-certs\") pod \"2e4b65a7-462d-4831-8cca-65f29dac08df\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") "
Nov 02 10:04:56.303476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.303019    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-podinfo\") pod \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") "
Nov 02 10:04:56.303476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.303053    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-token\" (UniqueName: \"kubernetes.io/projected/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-token\") pod \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") "
Nov 02 10:04:56.303476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.303104    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-envoy\") pod \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") "
Nov 02 10:04:56.303476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.303134    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istiod-ca-cert\") pod \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\" (UID: \"4c42817a-b1ca-486d-8f46-3db3bbf9346c\") "
Nov 02 10:04:56.303476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.303166    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-workload-socket\") pod \"2e4b65a7-462d-4831-8cca-65f29dac08df\" (UID: \"2e4b65a7-462d-4831-8cca-65f29dac08df\") "
Nov 02 10:04:56.303476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.302927    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-credential-socket" pod "2e4b65a7-462d-4831-8cca-65f29dac08df" (UID: "2e4b65a7-462d-4831-8cca-65f29dac08df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.303476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.303031    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-workload-certs" pod "4c42817a-b1ca-486d-8f46-3db3bbf9346c" (UID: "4c42817a-b1ca-486d-8f46-3db3bbf9346c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.303476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.303137    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-credential-socket" pod "4c42817a-b1ca-486d-8f46-3db3bbf9346c" (UID: "4c42817a-b1ca-486d-8f46-3db3bbf9346c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.304002 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.303281    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-workload-socket" pod "4c42817a-b1ca-486d-8f46-3db3bbf9346c" (UID: "4c42817a-b1ca-486d-8f46-3db3bbf9346c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.304002 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.303284    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4b65a7-462d-4831-8cca-65f29dac08df-istiod-ca-cert" pod "2e4b65a7-462d-4831-8cca-65f29dac08df" (UID: "2e4b65a7-462d-4831-8cca-65f29dac08df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:04:56.304002 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.303421    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-workload-certs" pod "2e4b65a7-462d-4831-8cca-65f29dac08df" (UID: "2e4b65a7-462d-4831-8cca-65f29dac08df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.304002 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.303617    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-data" pod "4c42817a-b1ca-486d-8f46-3db3bbf9346c" (UID: "4c42817a-b1ca-486d-8f46-3db3bbf9346c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.305452 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305429    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c42817a-b1ca-486d-8f46-3db3bbf9346c-kube-api-access-74kc2" pod "4c42817a-b1ca-486d-8f46-3db3bbf9346c" (UID: "4c42817a-b1ca-486d-8f46-3db3bbf9346c"). InnerVolumeSpecName "kube-api-access-74kc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:04:56.305578 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305464    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4b65a7-462d-4831-8cca-65f29dac08df-kube-api-access-699xx" pod "2e4b65a7-462d-4831-8cca-65f29dac08df" (UID: "2e4b65a7-462d-4831-8cca-65f29dac08df"). InnerVolumeSpecName "kube-api-access-699xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:04:56.305646 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305567    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-token" pod "2e4b65a7-462d-4831-8cca-65f29dac08df" (UID: "2e4b65a7-462d-4831-8cca-65f29dac08df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:04:56.305709 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305656    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-workload-socket" pod "2e4b65a7-462d-4831-8cca-65f29dac08df" (UID: "2e4b65a7-462d-4831-8cca-65f29dac08df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.305857 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305838    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.305912 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305865    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74kc2\" (UniqueName: \"kubernetes.io/projected/4c42817a-b1ca-486d-8f46-3db3bbf9346c-kube-api-access-74kc2\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.305912 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305868    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istiod-ca-cert" pod "4c42817a-b1ca-486d-8f46-3db3bbf9346c" (UID: "4c42817a-b1ca-486d-8f46-3db3bbf9346c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:04:56.305912 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305882    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.305912 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305898    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.306096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305912    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-699xx\" (UniqueName: \"kubernetes.io/projected/2e4b65a7-462d-4831-8cca-65f29dac08df-kube-api-access-699xx\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.306096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305932    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.306096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305945    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.306096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305960    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.306096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305974    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.306096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.305987    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e4b65a7-462d-4831-8cca-65f29dac08df-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.306096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.306000    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.306096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.306013    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.306458 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.306202    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-envoy" pod "4c42817a-b1ca-486d-8f46-3db3bbf9346c" (UID: "4c42817a-b1ca-486d-8f46-3db3bbf9346c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.306458 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.306208    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-envoy" pod "2e4b65a7-462d-4831-8cca-65f29dac08df" (UID: "2e4b65a7-462d-4831-8cca-65f29dac08df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:04:56.306458 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.306297    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-podinfo" pod "4c42817a-b1ca-486d-8f46-3db3bbf9346c" (UID: "4c42817a-b1ca-486d-8f46-3db3bbf9346c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:04:56.306458 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.306310    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-token" pod "4c42817a-b1ca-486d-8f46-3db3bbf9346c" (UID: "4c42817a-b1ca-486d-8f46-3db3bbf9346c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:04:56.306619 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.306599    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-podinfo" pod "2e4b65a7-462d-4831-8cca-65f29dac08df" (UID: "2e4b65a7-462d-4831-8cca-65f29dac08df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:04:56.406752 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.406716    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.406752 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.406746    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e4b65a7-462d-4831-8cca-65f29dac08df-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.406948 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.406760    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.406948 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.406771    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.406948 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.406780    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.406948 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.406788    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c42817a-b1ca-486d-8f46-3db3bbf9346c-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:04:56.847756 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.847719    4614 generic.go:356] "Generic (PLEG): container finished" podID="2e4b65a7-462d-4831-8cca-65f29dac08df" containerID="d11e1e11e374f42839eaa4e03c5b96e2a6e3d5ec2505793c0e6e81112b9177ac" exitCode=0
Nov 02 10:04:56.847964 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.847770    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"
Nov 02 10:04:56.847964 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.847791    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx" event={"ID":"2e4b65a7-462d-4831-8cca-65f29dac08df","Type":"ContainerDied","Data":"d11e1e11e374f42839eaa4e03c5b96e2a6e3d5ec2505793c0e6e81112b9177ac"}
Nov 02 10:04:56.847964 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.847857    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx" event={"ID":"2e4b65a7-462d-4831-8cca-65f29dac08df","Type":"ContainerDied","Data":"c60160da7df2a6bf2e07b83f8f7407389ad53ccaa0d605226ed2a4af60249f5b"}
Nov 02 10:04:56.847964 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.847887    4614 scope.go:122] "RemoveContainer" containerID="d11e1e11e374f42839eaa4e03c5b96e2a6e3d5ec2505793c0e6e81112b9177ac"
Nov 02 10:04:56.849424 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.849397    4614 generic.go:356] "Generic (PLEG): container finished" podID="4c42817a-b1ca-486d-8f46-3db3bbf9346c" containerID="06f7f1fa25c65eead796e403001245f5692e34630823f7ed59a085f14bc3ec0a" exitCode=0
Nov 02 10:04:56.849569 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.849436    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"
Nov 02 10:04:56.849569 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.849454    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn" event={"ID":"4c42817a-b1ca-486d-8f46-3db3bbf9346c","Type":"ContainerDied","Data":"06f7f1fa25c65eead796e403001245f5692e34630823f7ed59a085f14bc3ec0a"}
Nov 02 10:04:56.849569 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.849492    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn" event={"ID":"4c42817a-b1ca-486d-8f46-3db3bbf9346c","Type":"ContainerDied","Data":"463c062b720a2e91e84ddf25f829235714239a99dcda7e42ca686dce5918aec5"}
Nov 02 10:04:56.859006 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.858983    4614 scope.go:122] "RemoveContainer" containerID="d11e1e11e374f42839eaa4e03c5b96e2a6e3d5ec2505793c0e6e81112b9177ac"
Nov 02 10:04:56.859291 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:04:56.859274    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d11e1e11e374f42839eaa4e03c5b96e2a6e3d5ec2505793c0e6e81112b9177ac\": container with ID starting with d11e1e11e374f42839eaa4e03c5b96e2a6e3d5ec2505793c0e6e81112b9177ac not found: ID does not exist" containerID="d11e1e11e374f42839eaa4e03c5b96e2a6e3d5ec2505793c0e6e81112b9177ac"
Nov 02 10:04:56.859380 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.859300    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d11e1e11e374f42839eaa4e03c5b96e2a6e3d5ec2505793c0e6e81112b9177ac"} err="failed to get container status \"d11e1e11e374f42839eaa4e03c5b96e2a6e3d5ec2505793c0e6e81112b9177ac\": rpc error: code = NotFound desc = could not find container \"d11e1e11e374f42839eaa4e03c5b96e2a6e3d5ec2505793c0e6e81112b9177ac\": container with ID starting with d11e1e11e374f42839eaa4e03c5b96e2a6e3d5ec2505793c0e6e81112b9177ac not found: ID does not exist"
Nov 02 10:04:56.859380 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.859316    4614 scope.go:122] "RemoveContainer" containerID="06f7f1fa25c65eead796e403001245f5692e34630823f7ed59a085f14bc3ec0a"
Nov 02 10:04:56.868299 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.868279    4614 scope.go:122] "RemoveContainer" containerID="06f7f1fa25c65eead796e403001245f5692e34630823f7ed59a085f14bc3ec0a"
Nov 02 10:04:56.868569 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:04:56.868548    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f7f1fa25c65eead796e403001245f5692e34630823f7ed59a085f14bc3ec0a\": container with ID starting with 06f7f1fa25c65eead796e403001245f5692e34630823f7ed59a085f14bc3ec0a not found: ID does not exist" containerID="06f7f1fa25c65eead796e403001245f5692e34630823f7ed59a085f14bc3ec0a"
Nov 02 10:04:56.868644 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.868578    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f7f1fa25c65eead796e403001245f5692e34630823f7ed59a085f14bc3ec0a"} err="failed to get container status \"06f7f1fa25c65eead796e403001245f5692e34630823f7ed59a085f14bc3ec0a\": rpc error: code = NotFound desc = could not find container \"06f7f1fa25c65eead796e403001245f5692e34630823f7ed59a085f14bc3ec0a\": container with ID starting with 06f7f1fa25c65eead796e403001245f5692e34630823f7ed59a085f14bc3ec0a not found: ID does not exist"
Nov 02 10:04:56.873578 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.873548    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"]
Nov 02 10:04:56.879465 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.879442    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-cvdnx"]
Nov 02 10:04:56.890883 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.890849    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"]
Nov 02 10:04:56.893133 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:56.893112    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-629129-openshift-default-dc77fd7d9-wzxjn"]
Nov 02 10:04:57.315082 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:57.315048    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4b65a7-462d-4831-8cca-65f29dac08df" path="/var/lib/kubelet/pods/2e4b65a7-462d-4831-8cca-65f29dac08df/volumes"
Nov 02 10:04:57.315585 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:57.315571    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c42817a-b1ca-486d-8f46-3db3bbf9346c" path="/var/lib/kubelet/pods/4c42817a-b1ca-486d-8f46-3db3bbf9346c/volumes"
Nov 02 10:04:57.534448 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:57.534411    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"]
Nov 02 10:04:57.552745 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:04:57.552712    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"]
Nov 02 10:05:02.535623 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:02.535583    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr" podUID="3f19d824-07ba-46e9-b281-edd1a2dba5fd" containerName="payload-processing" containerID="cri-o://66c4c1bfc2cad55b6c9f3525c6dec536b0214a463c09f356993c131543d9855a" gracePeriod=25
Nov 02 10:05:02.553918 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:02.553884    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7" podUID="ebf91158-4d01-4d00-8b19-84b4e0e48064" containerName="payload-pre-processing" containerID="cri-o://141bea154f7f4c908daed0373ad7c2c87760e7c09d754257d522e48565b5b300" gracePeriod=25
Nov 02 10:05:05.686254 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:05.686216    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"]
Nov 02 10:05:05.686725 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:05.686534    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr" podUID="53d382ba-ee1f-4ec7-9b70-c2193c333d3e" containerName="istio-proxy" containerID="cri-o://6b0097d0a2d8bc97b7abdb33305fd2d4f723ffdd51f2d6e9e141ba3ff7c5e395" gracePeriod=30
Nov 02 10:05:05.697282 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:05.697252    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"]
Nov 02 10:05:05.697481 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:05.697459    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz" podUID="1b9e3aee-54bc-4bb1-8aa1-98b357aac526" containerName="istio-proxy" containerID="cri-o://572d6d832b0557eee710e9402f08782b79d7550b7a382b747226af61fabd7de1" gracePeriod=30
Nov 02 10:05:05.703597 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:05.703566    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"]
Nov 02 10:05:05.703793 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:05.703773    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz" podUID="c2222887-beb2-42c3-9404-7940c7349123" containerName="istio-proxy" containerID="cri-o://30581a200cacd46b21c726038a6e7d9597c0ef9bc7b3ea651db917971250d520" gracePeriod=30
Nov 02 10:05:09.580949 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:09.580919    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cfpwn_b0b53ec3-a9e4-45d7-8aa4-59237abc2614/manager/0.log"
Nov 02 10:05:09.635486 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:05:09.635452    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:05:09.963044 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:09.962969    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-j8qxd_7f946e8c-3ccb-4967-9d2d-61f67abb41ed/manager/0.log"
Nov 02 10:05:10.339017 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.338982    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b555b5c85-jv2xg_43fcb0a2-7557-4805-aff1-ac13b3ba44e4/manager/0.log"
Nov 02 10:05:10.952534 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.952162    4614 generic.go:356] "Generic (PLEG): container finished" podID="53d382ba-ee1f-4ec7-9b70-c2193c333d3e" containerID="6b0097d0a2d8bc97b7abdb33305fd2d4f723ffdd51f2d6e9e141ba3ff7c5e395" exitCode=0
Nov 02 10:05:10.952534 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.952374    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr" event={"ID":"53d382ba-ee1f-4ec7-9b70-c2193c333d3e","Type":"ContainerDied","Data":"6b0097d0a2d8bc97b7abdb33305fd2d4f723ffdd51f2d6e9e141ba3ff7c5e395"}
Nov 02 10:05:10.952534 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.952470    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr" event={"ID":"53d382ba-ee1f-4ec7-9b70-c2193c333d3e","Type":"ContainerDied","Data":"29cdf7df203e9d93bfc4a6f2fe4dde4f98b7ed0a8f923aacfe4171d2df443d51"}
Nov 02 10:05:10.952534 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.952496    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29cdf7df203e9d93bfc4a6f2fe4dde4f98b7ed0a8f923aacfe4171d2df443d51"
Nov 02 10:05:10.954476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.954428    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:05:10.955789 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.955764    4614 generic.go:356] "Generic (PLEG): container finished" podID="c2222887-beb2-42c3-9404-7940c7349123" containerID="30581a200cacd46b21c726038a6e7d9597c0ef9bc7b3ea651db917971250d520" exitCode=0
Nov 02 10:05:10.955885 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.955839    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz" event={"ID":"c2222887-beb2-42c3-9404-7940c7349123","Type":"ContainerDied","Data":"30581a200cacd46b21c726038a6e7d9597c0ef9bc7b3ea651db917971250d520"}
Nov 02 10:05:10.955885 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.955851    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7" podUID="ebf91158-4d01-4d00-8b19-84b4e0e48064" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 10:05:10.957548 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.957517    4614 generic.go:356] "Generic (PLEG): container finished" podID="1b9e3aee-54bc-4bb1-8aa1-98b357aac526" containerID="572d6d832b0557eee710e9402f08782b79d7550b7a382b747226af61fabd7de1" exitCode=0
Nov 02 10:05:10.957637 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.957566    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz" event={"ID":"1b9e3aee-54bc-4bb1-8aa1-98b357aac526","Type":"ContainerDied","Data":"572d6d832b0557eee710e9402f08782b79d7550b7a382b747226af61fabd7de1"}
Nov 02 10:05:10.968411 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.968384    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-token\" (UniqueName: \"kubernetes.io/projected/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-token\") pod \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") "
Nov 02 10:05:10.968588 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.968443    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-kube-api-access-4pxrx\" (UniqueName: \"kubernetes.io/projected/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-kube-api-access-4pxrx\") pod \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") "
Nov 02 10:05:10.968892 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.968853    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-credential-socket\") pod \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") "
Nov 02 10:05:10.969014 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.968928    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-workload-certs\") pod \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") "
Nov 02 10:05:10.969014 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.968985    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-podinfo\") pod \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") "
Nov 02 10:05:10.969137 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.969045    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-data\") pod \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") "
Nov 02 10:05:10.969137 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.969082    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-workload-socket\") pod \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") "
Nov 02 10:05:10.969137 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.969114    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istiod-ca-cert\") pod \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") "
Nov 02 10:05:10.969296 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.969163    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-envoy\") pod \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\" (UID: \"53d382ba-ee1f-4ec7-9b70-c2193c333d3e\") "
Nov 02 10:05:10.970441 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.970083    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-workload-certs" pod "53d382ba-ee1f-4ec7-9b70-c2193c333d3e" (UID: "53d382ba-ee1f-4ec7-9b70-c2193c333d3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:10.970441 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.970298    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-data" pod "53d382ba-ee1f-4ec7-9b70-c2193c333d3e" (UID: "53d382ba-ee1f-4ec7-9b70-c2193c333d3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:10.970441 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.970333    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istiod-ca-cert" pod "53d382ba-ee1f-4ec7-9b70-c2193c333d3e" (UID: "53d382ba-ee1f-4ec7-9b70-c2193c333d3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:10.970441 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.970379    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-credential-socket" pod "53d382ba-ee1f-4ec7-9b70-c2193c333d3e" (UID: "53d382ba-ee1f-4ec7-9b70-c2193c333d3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:10.970727 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.970518    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-workload-socket" pod "53d382ba-ee1f-4ec7-9b70-c2193c333d3e" (UID: "53d382ba-ee1f-4ec7-9b70-c2193c333d3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:10.971655 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.971632    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-kube-api-access-4pxrx" pod "53d382ba-ee1f-4ec7-9b70-c2193c333d3e" (UID: "53d382ba-ee1f-4ec7-9b70-c2193c333d3e"). InnerVolumeSpecName "kube-api-access-4pxrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:10.974468 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.973077    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-token" pod "53d382ba-ee1f-4ec7-9b70-c2193c333d3e" (UID: "53d382ba-ee1f-4ec7-9b70-c2193c333d3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:10.975050 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.975021    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-envoy" pod "53d382ba-ee1f-4ec7-9b70-c2193c333d3e" (UID: "53d382ba-ee1f-4ec7-9b70-c2193c333d3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:10.976938 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:10.976912    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-podinfo" pod "53d382ba-ee1f-4ec7-9b70-c2193c333d3e" (UID: "53d382ba-ee1f-4ec7-9b70-c2193c333d3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:05:11.005386 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.005363    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:05:11.013010 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.012990    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:05:11.069992 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.069952    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istiod-ca-cert\") pod \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") "
Nov 02 10:05:11.070175 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070016    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-workload-socket\") pod \"c2222887-beb2-42c3-9404-7940c7349123\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") "
Nov 02 10:05:11.070175 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070048    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-workload-certs\") pod \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") "
Nov 02 10:05:11.070175 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070161    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-podinfo\") pod \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070213    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-token\" (UniqueName: \"kubernetes.io/projected/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-token\") pod \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070257    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-workload-certs\") pod \"c2222887-beb2-42c3-9404-7940c7349123\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070299    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-workload-socket\") pod \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070315    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-workload-socket" pod "c2222887-beb2-42c3-9404-7940c7349123" (UID: "c2222887-beb2-42c3-9404-7940c7349123"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070366    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2222887-beb2-42c3-9404-7940c7349123-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2222887-beb2-42c3-9404-7940c7349123-istiod-ca-cert\") pod \"c2222887-beb2-42c3-9404-7940c7349123\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070347    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-workload-certs" pod "1b9e3aee-54bc-4bb1-8aa1-98b357aac526" (UID: "1b9e3aee-54bc-4bb1-8aa1-98b357aac526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070409    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-credential-socket\") pod \"c2222887-beb2-42c3-9404-7940c7349123\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070446    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-istio-data\") pod \"c2222887-beb2-42c3-9404-7940c7349123\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070466    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istiod-ca-cert" pod "1b9e3aee-54bc-4bb1-8aa1-98b357aac526" (UID: "1b9e3aee-54bc-4bb1-8aa1-98b357aac526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070484    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-envoy\") pod \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070519    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-data\") pod \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070544    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2222887-beb2-42c3-9404-7940c7349123-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2222887-beb2-42c3-9404-7940c7349123-istio-podinfo\") pod \"c2222887-beb2-42c3-9404-7940c7349123\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070575    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2222887-beb2-42c3-9404-7940c7349123-istio-token\" (UniqueName: \"kubernetes.io/projected/c2222887-beb2-42c3-9404-7940c7349123-istio-token\") pod \"c2222887-beb2-42c3-9404-7940c7349123\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070602    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-workload-certs" pod "c2222887-beb2-42c3-9404-7940c7349123" (UID: "c2222887-beb2-42c3-9404-7940c7349123"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070613    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-istio-envoy\") pod \"c2222887-beb2-42c3-9404-7940c7349123\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070616    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-workload-socket" pod "1b9e3aee-54bc-4bb1-8aa1-98b357aac526" (UID: "1b9e3aee-54bc-4bb1-8aa1-98b357aac526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070629    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-credential-socket" pod "c2222887-beb2-42c3-9404-7940c7349123" (UID: "c2222887-beb2-42c3-9404-7940c7349123"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070651    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-kube-api-access-b82q5\" (UniqueName: \"kubernetes.io/projected/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-kube-api-access-b82q5\") pod \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070694    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-credential-socket\") pod \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\" (UID: \"1b9e3aee-54bc-4bb1-8aa1-98b357aac526\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070725    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2222887-beb2-42c3-9404-7940c7349123-kube-api-access-2llk5\" (UniqueName: \"kubernetes.io/projected/c2222887-beb2-42c3-9404-7940c7349123-kube-api-access-2llk5\") pod \"c2222887-beb2-42c3-9404-7940c7349123\" (UID: \"c2222887-beb2-42c3-9404-7940c7349123\") "
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070783    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-istio-data" pod "c2222887-beb2-42c3-9404-7940c7349123" (UID: "c2222887-beb2-42c3-9404-7940c7349123"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.070993 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.070790    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-data" pod "1b9e3aee-54bc-4bb1-8aa1-98b357aac526" (UID: "1b9e3aee-54bc-4bb1-8aa1-98b357aac526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071066    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071089    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071105    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071118    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071134    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071147    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pxrx\" (UniqueName: \"kubernetes.io/projected/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-kube-api-access-4pxrx\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071161    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071176    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071189    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071202    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071216    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071230    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071244    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071258    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071273    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071286    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d382ba-ee1f-4ec7-9b70-c2193c333d3e-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071300    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071403    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-credential-socket" pod "1b9e3aee-54bc-4bb1-8aa1-98b357aac526" (UID: "1b9e3aee-54bc-4bb1-8aa1-98b357aac526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.072396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.071467    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2222887-beb2-42c3-9404-7940c7349123-istiod-ca-cert" pod "c2222887-beb2-42c3-9404-7940c7349123" (UID: "c2222887-beb2-42c3-9404-7940c7349123"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:11.073729 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.073695    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-podinfo" pod "1b9e3aee-54bc-4bb1-8aa1-98b357aac526" (UID: "1b9e3aee-54bc-4bb1-8aa1-98b357aac526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:05:11.073828 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.073738    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-kube-api-access-b82q5" pod "1b9e3aee-54bc-4bb1-8aa1-98b357aac526" (UID: "1b9e3aee-54bc-4bb1-8aa1-98b357aac526"). InnerVolumeSpecName "kube-api-access-b82q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:11.073828 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.073747    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-envoy" pod "1b9e3aee-54bc-4bb1-8aa1-98b357aac526" (UID: "1b9e3aee-54bc-4bb1-8aa1-98b357aac526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.073828 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.073751    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2222887-beb2-42c3-9404-7940c7349123-istio-podinfo" pod "c2222887-beb2-42c3-9404-7940c7349123" (UID: "c2222887-beb2-42c3-9404-7940c7349123"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:05:11.073999 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.073867    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2222887-beb2-42c3-9404-7940c7349123-kube-api-access-2llk5" pod "c2222887-beb2-42c3-9404-7940c7349123" (UID: "c2222887-beb2-42c3-9404-7940c7349123"). InnerVolumeSpecName "kube-api-access-2llk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:11.075020 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.074990    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2222887-beb2-42c3-9404-7940c7349123-istio-token" pod "c2222887-beb2-42c3-9404-7940c7349123" (UID: "c2222887-beb2-42c3-9404-7940c7349123"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:11.075132 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.075064    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-token" pod "1b9e3aee-54bc-4bb1-8aa1-98b357aac526" (UID: "1b9e3aee-54bc-4bb1-8aa1-98b357aac526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:11.075199 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.075164    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-istio-envoy" pod "c2222887-beb2-42c3-9404-7940c7349123" (UID: "c2222887-beb2-42c3-9404-7940c7349123"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:11.171884 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.171851    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.171884 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.171881    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.171884 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.171890    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2222887-beb2-42c3-9404-7940c7349123-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.172120 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.171899    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.172120 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.171908    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2222887-beb2-42c3-9404-7940c7349123-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.172120 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.171917    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2222887-beb2-42c3-9404-7940c7349123-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.172120 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.171925    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2222887-beb2-42c3-9404-7940c7349123-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.172120 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.171935    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b82q5\" (UniqueName: \"kubernetes.io/projected/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-kube-api-access-b82q5\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.172120 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.171944    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9e3aee-54bc-4bb1-8aa1-98b357aac526-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.172120 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.171953    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2llk5\" (UniqueName: \"kubernetes.io/projected/c2222887-beb2-42c3-9404-7940c7349123-kube-api-access-2llk5\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:11.954117 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.954030    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr" podUID="3f19d824-07ba-46e9-b281-edd1a2dba5fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 10:05:11.963826 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.963800    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"
Nov 02 10:05:11.963826 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.963809    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz" event={"ID":"c2222887-beb2-42c3-9404-7940c7349123","Type":"ContainerDied","Data":"e48577d6b0c3678d0b2c973437dc812afca0d1da65c7a26a4690c511c632c27a"}
Nov 02 10:05:11.964054 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.963877    4614 scope.go:122] "RemoveContainer" containerID="30581a200cacd46b21c726038a6e7d9597c0ef9bc7b3ea651db917971250d520"
Nov 02 10:05:11.965420 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.965393    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz" event={"ID":"1b9e3aee-54bc-4bb1-8aa1-98b357aac526","Type":"ContainerDied","Data":"4556c5714827efefb04d3a6071325284b2a9b26b628da25b0d49b5a3ef21a630"}
Nov 02 10:05:11.965520 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.965425    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"
Nov 02 10:05:11.965520 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.965435    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"
Nov 02 10:05:11.973491 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.973476    4614 scope.go:122] "RemoveContainer" containerID="572d6d832b0557eee710e9402f08782b79d7550b7a382b747226af61fabd7de1"
Nov 02 10:05:11.989273 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.989250    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"]
Nov 02 10:05:11.993753 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:11.993725    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-sn5sr"]
Nov 02 10:05:12.005067 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:12.005046    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"]
Nov 02 10:05:12.010550 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:12.010528    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-mfjwz"]
Nov 02 10:05:12.022021 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:12.022000    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"]
Nov 02 10:05:12.027498 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:12.027472    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-204fba-openshift-default-8dd6555f7-7glvz"]
Nov 02 10:05:13.314985 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:13.314948    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b9e3aee-54bc-4bb1-8aa1-98b357aac526" path="/var/lib/kubelet/pods/1b9e3aee-54bc-4bb1-8aa1-98b357aac526/volumes"
Nov 02 10:05:13.315402 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:13.315387    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53d382ba-ee1f-4ec7-9b70-c2193c333d3e" path="/var/lib/kubelet/pods/53d382ba-ee1f-4ec7-9b70-c2193c333d3e/volumes"
Nov 02 10:05:13.315787 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:13.315775    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2222887-beb2-42c3-9404-7940c7349123" path="/var/lib/kubelet/pods/c2222887-beb2-42c3-9404-7940c7349123/volumes"
Nov 02 10:05:19.799165 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:19.799130    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:05:19.821583 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:19.821551    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:05:20.957448 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:20.957412    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7" podUID="ebf91158-4d01-4d00-8b19-84b4e0e48064" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 10:05:21.953979 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:21.953941    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr" podUID="3f19d824-07ba-46e9-b281-edd1a2dba5fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 10:05:22.811120 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:22.811086    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"
Nov 02 10:05:22.831511 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:22.831476    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"
Nov 02 10:05:22.873477 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:22.873449    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf91158-4d01-4d00-8b19-84b4e0e48064-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf91158-4d01-4d00-8b19-84b4e0e48064-plugins-config-volume\") pod \"ebf91158-4d01-4d00-8b19-84b4e0e48064\" (UID: \"ebf91158-4d01-4d00-8b19-84b4e0e48064\") "
Nov 02 10:05:22.873630 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:22.873533    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf91158-4d01-4d00-8b19-84b4e0e48064-kube-api-access-745pv\" (UniqueName: \"kubernetes.io/projected/ebf91158-4d01-4d00-8b19-84b4e0e48064-kube-api-access-745pv\") pod \"ebf91158-4d01-4d00-8b19-84b4e0e48064\" (UID: \"ebf91158-4d01-4d00-8b19-84b4e0e48064\") "
Nov 02 10:05:22.873630 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:22.873557    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f19d824-07ba-46e9-b281-edd1a2dba5fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f19d824-07ba-46e9-b281-edd1a2dba5fd-plugins-config-volume\") pod \"3f19d824-07ba-46e9-b281-edd1a2dba5fd\" (UID: \"3f19d824-07ba-46e9-b281-edd1a2dba5fd\") "
Nov 02 10:05:22.873630 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:22.873605    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f19d824-07ba-46e9-b281-edd1a2dba5fd-kube-api-access-sgwbq\" (UniqueName: \"kubernetes.io/projected/3f19d824-07ba-46e9-b281-edd1a2dba5fd-kube-api-access-sgwbq\") pod \"3f19d824-07ba-46e9-b281-edd1a2dba5fd\" (UID: \"3f19d824-07ba-46e9-b281-edd1a2dba5fd\") "
Nov 02 10:05:22.873885 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:22.873858    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf91158-4d01-4d00-8b19-84b4e0e48064-plugins-config-volume" pod "ebf91158-4d01-4d00-8b19-84b4e0e48064" (UID: "ebf91158-4d01-4d00-8b19-84b4e0e48064"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:22.873991 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:22.873943    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f19d824-07ba-46e9-b281-edd1a2dba5fd-plugins-config-volume" pod "3f19d824-07ba-46e9-b281-edd1a2dba5fd" (UID: "3f19d824-07ba-46e9-b281-edd1a2dba5fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:22.875731 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:22.875705    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf91158-4d01-4d00-8b19-84b4e0e48064-kube-api-access-745pv" pod "ebf91158-4d01-4d00-8b19-84b4e0e48064" (UID: "ebf91158-4d01-4d00-8b19-84b4e0e48064"). InnerVolumeSpecName "kube-api-access-745pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:22.875817 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:22.875791    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f19d824-07ba-46e9-b281-edd1a2dba5fd-kube-api-access-sgwbq" pod "3f19d824-07ba-46e9-b281-edd1a2dba5fd" (UID: "3f19d824-07ba-46e9-b281-edd1a2dba5fd"). InnerVolumeSpecName "kube-api-access-sgwbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:22.974808 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:22.974773    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf91158-4d01-4d00-8b19-84b4e0e48064-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:22.974808 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:22.974802    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-745pv\" (UniqueName: \"kubernetes.io/projected/ebf91158-4d01-4d00-8b19-84b4e0e48064-kube-api-access-745pv\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:22.974808 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:22.974812    4614 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f19d824-07ba-46e9-b281-edd1a2dba5fd-plugins-config-volume\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:22.975054 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:22.974820    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgwbq\" (UniqueName: \"kubernetes.io/projected/3f19d824-07ba-46e9-b281-edd1a2dba5fd-kube-api-access-sgwbq\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:23.043196 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.043164    4614 generic.go:356] "Generic (PLEG): container finished" podID="3f19d824-07ba-46e9-b281-edd1a2dba5fd" containerID="66c4c1bfc2cad55b6c9f3525c6dec536b0214a463c09f356993c131543d9855a" exitCode=0
Nov 02 10:05:23.043390 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.043208    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"
Nov 02 10:05:23.043390 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.043242    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr" event={"ID":"3f19d824-07ba-46e9-b281-edd1a2dba5fd","Type":"ContainerDied","Data":"66c4c1bfc2cad55b6c9f3525c6dec536b0214a463c09f356993c131543d9855a"}
Nov 02 10:05:23.043390 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.043334    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr" event={"ID":"3f19d824-07ba-46e9-b281-edd1a2dba5fd","Type":"ContainerDied","Data":"16cbe7b72a823511e1d614090f1bc3f427fef3e26fb30a8573fd1ed197cfed35"}
Nov 02 10:05:23.043390 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.043385    4614 scope.go:122] "RemoveContainer" containerID="66c4c1bfc2cad55b6c9f3525c6dec536b0214a463c09f356993c131543d9855a"
Nov 02 10:05:23.044856 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.044786    4614 generic.go:356] "Generic (PLEG): container finished" podID="ebf91158-4d01-4d00-8b19-84b4e0e48064" containerID="141bea154f7f4c908daed0373ad7c2c87760e7c09d754257d522e48565b5b300" exitCode=0
Nov 02 10:05:23.044856 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.044824    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7" event={"ID":"ebf91158-4d01-4d00-8b19-84b4e0e48064","Type":"ContainerDied","Data":"141bea154f7f4c908daed0373ad7c2c87760e7c09d754257d522e48565b5b300"}
Nov 02 10:05:23.044856 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.044853    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7" event={"ID":"ebf91158-4d01-4d00-8b19-84b4e0e48064","Type":"ContainerDied","Data":"a81ba73ab8d683ff183ac3d8b8c589eca1a65851210ec31835f396dec3095608"}
Nov 02 10:05:23.044996 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.044852    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"
Nov 02 10:05:23.053799 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.053782    4614 scope.go:122] "RemoveContainer" containerID="66c4c1bfc2cad55b6c9f3525c6dec536b0214a463c09f356993c131543d9855a"
Nov 02 10:05:23.054108 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:05:23.054087    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c4c1bfc2cad55b6c9f3525c6dec536b0214a463c09f356993c131543d9855a\": container with ID starting with 66c4c1bfc2cad55b6c9f3525c6dec536b0214a463c09f356993c131543d9855a not found: ID does not exist" containerID="66c4c1bfc2cad55b6c9f3525c6dec536b0214a463c09f356993c131543d9855a"
Nov 02 10:05:23.054206 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.054121    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c4c1bfc2cad55b6c9f3525c6dec536b0214a463c09f356993c131543d9855a"} err="failed to get container status \"66c4c1bfc2cad55b6c9f3525c6dec536b0214a463c09f356993c131543d9855a\": rpc error: code = NotFound desc = could not find container \"66c4c1bfc2cad55b6c9f3525c6dec536b0214a463c09f356993c131543d9855a\": container with ID starting with 66c4c1bfc2cad55b6c9f3525c6dec536b0214a463c09f356993c131543d9855a not found: ID does not exist"
Nov 02 10:05:23.054206 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.054146    4614 scope.go:122] "RemoveContainer" containerID="141bea154f7f4c908daed0373ad7c2c87760e7c09d754257d522e48565b5b300"
Nov 02 10:05:23.062524 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.062503    4614 scope.go:122] "RemoveContainer" containerID="141bea154f7f4c908daed0373ad7c2c87760e7c09d754257d522e48565b5b300"
Nov 02 10:05:23.063146 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:05:23.063121    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141bea154f7f4c908daed0373ad7c2c87760e7c09d754257d522e48565b5b300\": container with ID starting with 141bea154f7f4c908daed0373ad7c2c87760e7c09d754257d522e48565b5b300 not found: ID does not exist" containerID="141bea154f7f4c908daed0373ad7c2c87760e7c09d754257d522e48565b5b300"
Nov 02 10:05:23.063218 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.063154    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141bea154f7f4c908daed0373ad7c2c87760e7c09d754257d522e48565b5b300"} err="failed to get container status \"141bea154f7f4c908daed0373ad7c2c87760e7c09d754257d522e48565b5b300\": rpc error: code = NotFound desc = could not find container \"141bea154f7f4c908daed0373ad7c2c87760e7c09d754257d522e48565b5b300\": container with ID starting with 141bea154f7f4c908daed0373ad7c2c87760e7c09d754257d522e48565b5b300 not found: ID does not exist"
Nov 02 10:05:23.074372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.074331    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"]
Nov 02 10:05:23.076039 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.076016    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-204fba-558699b45f-np9qr"]
Nov 02 10:05:23.090815 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.090793    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"]
Nov 02 10:05:23.097541 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.097518    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-204fba-6786ccc979-d7ct7"]
Nov 02 10:05:23.314396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.314310    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f19d824-07ba-46e9-b281-edd1a2dba5fd" path="/var/lib/kubelet/pods/3f19d824-07ba-46e9-b281-edd1a2dba5fd/volumes"
Nov 02 10:05:23.314731 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:23.314715    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf91158-4d01-4d00-8b19-84b4e0e48064" path="/var/lib/kubelet/pods/ebf91158-4d01-4d00-8b19-84b4e0e48064/volumes"
Nov 02 10:05:34.601065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601033    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"]
Nov 02 10:05:34.601496 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601427    4614 cpu_manager.go:414] "containerMap: removing container" podUID="ceee9859-b557-4daa-8027-bf45e3d59e82" containerName="payload-processing"
Nov 02 10:05:34.601496 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601440    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceee9859-b557-4daa-8027-bf45e3d59e82" containerName="payload-processing"
Nov 02 10:05:34.601496 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601451    4614 cpu_manager.go:414] "containerMap: removing container" podUID="2e4b65a7-462d-4831-8cca-65f29dac08df" containerName="istio-proxy"
Nov 02 10:05:34.601496 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601456    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4b65a7-462d-4831-8cca-65f29dac08df" containerName="istio-proxy"
Nov 02 10:05:34.601496 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601465    4614 cpu_manager.go:414] "containerMap: removing container" podUID="c2222887-beb2-42c3-9404-7940c7349123" containerName="istio-proxy"
Nov 02 10:05:34.601496 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601471    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2222887-beb2-42c3-9404-7940c7349123" containerName="istio-proxy"
Nov 02 10:05:34.601496 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601478    4614 cpu_manager.go:414] "containerMap: removing container" podUID="1b9e3aee-54bc-4bb1-8aa1-98b357aac526" containerName="istio-proxy"
Nov 02 10:05:34.601496 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601483    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9e3aee-54bc-4bb1-8aa1-98b357aac526" containerName="istio-proxy"
Nov 02 10:05:34.601496 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601498    4614 cpu_manager.go:414] "containerMap: removing container" podUID="3f19d824-07ba-46e9-b281-edd1a2dba5fd" containerName="payload-processing"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601503    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f19d824-07ba-46e9-b281-edd1a2dba5fd" containerName="payload-processing"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601510    4614 cpu_manager.go:414] "containerMap: removing container" podUID="ebf91158-4d01-4d00-8b19-84b4e0e48064" containerName="payload-pre-processing"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601516    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf91158-4d01-4d00-8b19-84b4e0e48064" containerName="payload-pre-processing"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601531    4614 cpu_manager.go:414] "containerMap: removing container" podUID="53d382ba-ee1f-4ec7-9b70-c2193c333d3e" containerName="istio-proxy"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601535    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53d382ba-ee1f-4ec7-9b70-c2193c333d3e" containerName="istio-proxy"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601541    4614 cpu_manager.go:414] "containerMap: removing container" podUID="99ce62a8-8fd1-400f-8fb6-c81574f5ed65" containerName="payload-processing"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601546    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ce62a8-8fd1-400f-8fb6-c81574f5ed65" containerName="payload-processing"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601556    4614 cpu_manager.go:414] "containerMap: removing container" podUID="4c42817a-b1ca-486d-8f46-3db3bbf9346c" containerName="istio-proxy"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601574    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c42817a-b1ca-486d-8f46-3db3bbf9346c" containerName="istio-proxy"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601581    4614 cpu_manager.go:414] "containerMap: removing container" podUID="7e3ea645-ba8e-4ebb-8332-1617c13436be" containerName="maas-api"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601586    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3ea645-ba8e-4ebb-8332-1617c13436be" containerName="maas-api"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601641    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="53d382ba-ee1f-4ec7-9b70-c2193c333d3e" containerName="istio-proxy"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601648    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ce62a8-8fd1-400f-8fb6-c81574f5ed65" containerName="payload-processing"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601655    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c42817a-b1ca-486d-8f46-3db3bbf9346c" containerName="istio-proxy"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601661    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3ea645-ba8e-4ebb-8332-1617c13436be" containerName="maas-api"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601667    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceee9859-b557-4daa-8027-bf45e3d59e82" containerName="payload-processing"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601673    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4b65a7-462d-4831-8cca-65f29dac08df" containerName="istio-proxy"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601679    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2222887-beb2-42c3-9404-7940c7349123" containerName="istio-proxy"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601684    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9e3aee-54bc-4bb1-8aa1-98b357aac526" containerName="istio-proxy"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601691    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f19d824-07ba-46e9-b281-edd1a2dba5fd" containerName="payload-processing"
Nov 02 10:05:34.601833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.601696    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf91158-4d01-4d00-8b19-84b4e0e48064" containerName="payload-pre-processing"
Nov 02 10:05:34.605928 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.605909    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"
Nov 02 10:05:34.611455 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.611426    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:05:34.612281 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.612243    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2hk6k\""
Nov 02 10:05:34.615110 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.615083    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"]
Nov 02 10:05:34.694772 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.694734    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz8qg\" (UniqueName: \"kubernetes.io/projected/7bdc4594-88de-4f38-a86b-6c4d3e530932-kube-api-access-fz8qg\") pod \"maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf\" (UID: \"7bdc4594-88de-4f38-a86b-6c4d3e530932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"
Nov 02 10:05:34.694927 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.694787    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bdc4594-88de-4f38-a86b-6c4d3e530932-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf\" (UID: \"7bdc4594-88de-4f38-a86b-6c4d3e530932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"
Nov 02 10:05:34.694927 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.694899    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc4594-88de-4f38-a86b-6c4d3e530932-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf\" (UID: \"7bdc4594-88de-4f38-a86b-6c4d3e530932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"
Nov 02 10:05:34.796409 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.796370    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz8qg\" (UniqueName: \"kubernetes.io/projected/7bdc4594-88de-4f38-a86b-6c4d3e530932-kube-api-access-fz8qg\") pod \"maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf\" (UID: \"7bdc4594-88de-4f38-a86b-6c4d3e530932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"
Nov 02 10:05:34.796603 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.796423    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bdc4594-88de-4f38-a86b-6c4d3e530932-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf\" (UID: \"7bdc4594-88de-4f38-a86b-6c4d3e530932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"
Nov 02 10:05:34.796603 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.796462    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc4594-88de-4f38-a86b-6c4d3e530932-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf\" (UID: \"7bdc4594-88de-4f38-a86b-6c4d3e530932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"
Nov 02 10:05:34.797256 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.797228    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc4594-88de-4f38-a86b-6c4d3e530932-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf\" (UID: \"7bdc4594-88de-4f38-a86b-6c4d3e530932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"
Nov 02 10:05:34.799464 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:34.799346    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bdc4594-88de-4f38-a86b-6c4d3e530932-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf\" (UID: \"7bdc4594-88de-4f38-a86b-6c4d3e530932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"
Nov 02 10:05:34.802611 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:05:34.802586    4614 projected.go:196] Error preparing data for projected volume kube-api-access-fz8qg for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 02 10:05:34.802732 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:05:34.802660    4614 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bdc4594-88de-4f38-a86b-6c4d3e530932-kube-api-access-fz8qg podName:7bdc4594-88de-4f38-a86b-6c4d3e530932 nodeName:}" failed. No retries permitted until 2026-11-02 10:05:35.302640072 +0000 UTC m=+1816.135298143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fz8qg" (UniqueName: "kubernetes.io/projected/7bdc4594-88de-4f38-a86b-6c4d3e530932-kube-api-access-fz8qg") pod "maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf" (UID: "7bdc4594-88de-4f38-a86b-6c4d3e530932") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 02 10:05:35.402746 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:35.402712    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz8qg\" (UniqueName: \"kubernetes.io/projected/7bdc4594-88de-4f38-a86b-6c4d3e530932-kube-api-access-fz8qg\") pod \"maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf\" (UID: \"7bdc4594-88de-4f38-a86b-6c4d3e530932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"
Nov 02 10:05:35.415552 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:35.415517    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz8qg\" (UniqueName: \"kubernetes.io/projected/7bdc4594-88de-4f38-a86b-6c4d3e530932-kube-api-access-fz8qg\") pod \"maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf\" (UID: \"7bdc4594-88de-4f38-a86b-6c4d3e530932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"
Nov 02 10:05:35.520058 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:35.520017    4614 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf" secret="" err="secret \"maas-api-cleanup-dockercfg-2hk6k\" not found"
Nov 02 10:05:35.520276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:35.520085    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"
Nov 02 10:05:35.794115 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:35.794013    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"]
Nov 02 10:05:35.797979 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:05:35.797945    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdc4594_88de_4f38_a86b_6c4d3e530932.slice/crio-581140a66386a4b8e4b07698f85fcab2f860477122874c60b2cf15962457444a WatchSource:0}: Error finding container 581140a66386a4b8e4b07698f85fcab2f860477122874c60b2cf15962457444a: Status 404 returned error can't find the container with id 581140a66386a4b8e4b07698f85fcab2f860477122874c60b2cf15962457444a
Nov 02 10:05:36.119336 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:36.119301    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf" event={"ID":"7bdc4594-88de-4f38-a86b-6c4d3e530932","Type":"ContainerStarted","Data":"581140a66386a4b8e4b07698f85fcab2f860477122874c60b2cf15962457444a"}
Nov 02 10:05:37.125262 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:37.125223    4614 generic.go:356] "Generic (PLEG): container finished" podID="7bdc4594-88de-4f38-a86b-6c4d3e530932" containerID="3e076bf794f6b924ba6e56418efcb702d97adf10e8c83eac5ddbc7b3bdbbdac2" exitCode=0
Nov 02 10:05:37.125686 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:37.125299    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf" event={"ID":"7bdc4594-88de-4f38-a86b-6c4d3e530932","Type":"ContainerDied","Data":"3e076bf794f6b924ba6e56418efcb702d97adf10e8c83eac5ddbc7b3bdbbdac2"}
Nov 02 10:05:38.263552 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.263528    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"
Nov 02 10:05:38.339856 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.339826    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bdc4594-88de-4f38-a86b-6c4d3e530932-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bdc4594-88de-4f38-a86b-6c4d3e530932-maas-api-xks-ca\") pod \"7bdc4594-88de-4f38-a86b-6c4d3e530932\" (UID: \"7bdc4594-88de-4f38-a86b-6c4d3e530932\") "
Nov 02 10:05:38.340048 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.339867    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bdc4594-88de-4f38-a86b-6c4d3e530932-kube-api-access-fz8qg\" (UniqueName: \"kubernetes.io/projected/7bdc4594-88de-4f38-a86b-6c4d3e530932-kube-api-access-fz8qg\") pod \"7bdc4594-88de-4f38-a86b-6c4d3e530932\" (UID: \"7bdc4594-88de-4f38-a86b-6c4d3e530932\") "
Nov 02 10:05:38.340048 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.339933    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bdc4594-88de-4f38-a86b-6c4d3e530932-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc4594-88de-4f38-a86b-6c4d3e530932-maas-api-openshift-ca\") pod \"7bdc4594-88de-4f38-a86b-6c4d3e530932\" (UID: \"7bdc4594-88de-4f38-a86b-6c4d3e530932\") "
Nov 02 10:05:38.340499 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.340463    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bdc4594-88de-4f38-a86b-6c4d3e530932-maas-api-openshift-ca" pod "7bdc4594-88de-4f38-a86b-6c4d3e530932" (UID: "7bdc4594-88de-4f38-a86b-6c4d3e530932"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:38.342146 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.342119    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdc4594-88de-4f38-a86b-6c4d3e530932-kube-api-access-fz8qg" pod "7bdc4594-88de-4f38-a86b-6c4d3e530932" (UID: "7bdc4594-88de-4f38-a86b-6c4d3e530932"). InnerVolumeSpecName "kube-api-access-fz8qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:38.342243 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.342198    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bdc4594-88de-4f38-a86b-6c4d3e530932-maas-api-xks-ca" pod "7bdc4594-88de-4f38-a86b-6c4d3e530932" (UID: "7bdc4594-88de-4f38-a86b-6c4d3e530932"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:05:38.441818 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.441732    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bdc4594-88de-4f38-a86b-6c4d3e530932-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:38.441818 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.441765    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz8qg\" (UniqueName: \"kubernetes.io/projected/7bdc4594-88de-4f38-a86b-6c4d3e530932-kube-api-access-fz8qg\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:38.441818 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.441776    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc4594-88de-4f38-a86b-6c4d3e530932-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:38.964687 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.964654    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"]
Nov 02 10:05:38.964903 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.964881    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy" containerID="cri-o://619c08182293544fa8052b92d33463817c457f4fe02b637526beee3d12a216e6" gracePeriod=30
Nov 02 10:05:38.966676 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.966635    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"]
Nov 02 10:05:38.966938 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.966910    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy" containerID="cri-o://4f24b79772502bbe789ec2c9a215d33b16fc1fe37d00eab00950d645b7f11372" gracePeriod=30
Nov 02 10:05:38.968310 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.968284    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"]
Nov 02 10:05:38.968523 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:38.968494    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" podUID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" containerName="istio-proxy" containerID="cri-o://2f087b93cea8c27547aaa5a52530d4a14c04daf23ed726ab7b4a8400b42a1d3e" gracePeriod=30
Nov 02 10:05:39.135433 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:39.135396    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf" event={"ID":"7bdc4594-88de-4f38-a86b-6c4d3e530932","Type":"ContainerDied","Data":"581140a66386a4b8e4b07698f85fcab2f860477122874c60b2cf15962457444a"}
Nov 02 10:05:39.135433 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:39.135424    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"
Nov 02 10:05:39.135680 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:39.135447    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="581140a66386a4b8e4b07698f85fcab2f860477122874c60b2cf15962457444a"
Nov 02 10:05:39.636364 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:05:39.636314    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:05:44.177707 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.177672    4614 generic.go:356] "Generic (PLEG): container finished" podID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" containerID="2f087b93cea8c27547aaa5a52530d4a14c04daf23ed726ab7b4a8400b42a1d3e" exitCode=0
Nov 02 10:05:44.178165 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.178145    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" event={"ID":"6f527771-b4d1-43ee-89b4-264cfdf7f4d0","Type":"ContainerDied","Data":"2f087b93cea8c27547aaa5a52530d4a14c04daf23ed726ab7b4a8400b42a1d3e"}
Nov 02 10:05:44.180083 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.180064    4614 generic.go:356] "Generic (PLEG): container finished" podID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerID="619c08182293544fa8052b92d33463817c457f4fe02b637526beee3d12a216e6" exitCode=0
Nov 02 10:05:44.180243 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.180224    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" event={"ID":"125ab3c0-c75a-4866-bbfd-d490769e6146","Type":"ContainerDied","Data":"619c08182293544fa8052b92d33463817c457f4fe02b637526beee3d12a216e6"}
Nov 02 10:05:44.182481 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.182331    4614 generic.go:356] "Generic (PLEG): container finished" podID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerID="4f24b79772502bbe789ec2c9a215d33b16fc1fe37d00eab00950d645b7f11372" exitCode=0
Nov 02 10:05:44.182481 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.182387    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" event={"ID":"f0c328ac-256e-4e52-bb7b-81dfac9df62f","Type":"ContainerDied","Data":"4f24b79772502bbe789ec2c9a215d33b16fc1fe37d00eab00950d645b7f11372"}
Nov 02 10:05:44.233567 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.233544    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 10:05:44.273084 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.273053    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 10:05:44.307233 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.307204    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 10:05:44.400088 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400058    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-data\") pod \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") "
Nov 02 10:05:44.400272 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400096    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-envoy\") pod \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") "
Nov 02 10:05:44.400272 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400117    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-envoy\") pod \"125ab3c0-c75a-4866-bbfd-d490769e6146\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") "
Nov 02 10:05:44.400272 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400137    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-podinfo\") pod \"125ab3c0-c75a-4866-bbfd-d490769e6146\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") "
Nov 02 10:05:44.400272 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400183    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-credential-socket\") pod \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") "
Nov 02 10:05:44.400272 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400201    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-workload-socket\") pod \"125ab3c0-c75a-4866-bbfd-d490769e6146\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") "
Nov 02 10:05:44.400272 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400220    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-kube-api-access-zzzjg\" (UniqueName: \"kubernetes.io/projected/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-kube-api-access-zzzjg\") pod \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") "
Nov 02 10:05:44.400272 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400243    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125ab3c0-c75a-4866-bbfd-d490769e6146-kube-api-access-b9f8g\" (UniqueName: \"kubernetes.io/projected/125ab3c0-c75a-4866-bbfd-d490769e6146-kube-api-access-b9f8g\") pod \"125ab3c0-c75a-4866-bbfd-d490769e6146\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") "
Nov 02 10:05:44.400272 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400275    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-credential-socket\") pod \"125ab3c0-c75a-4866-bbfd-d490769e6146\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") "
Nov 02 10:05:44.400752 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400325    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-workload-socket\") pod \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") "
Nov 02 10:05:44.400752 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400374    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-data" pod "6f527771-b4d1-43ee-89b4-264cfdf7f4d0" (UID: "6f527771-b4d1-43ee-89b4-264cfdf7f4d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.400752 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400388    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-workload-certs\") pod \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") "
Nov 02 10:05:44.400752 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400427    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-workload-socket\") pod \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") "
Nov 02 10:05:44.400752 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400480    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-podinfo\") pod \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") "
Nov 02 10:05:44.400752 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400487    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-credential-socket" pod "f0c328ac-256e-4e52-bb7b-81dfac9df62f" (UID: "f0c328ac-256e-4e52-bb7b-81dfac9df62f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.400752 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400513    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-data\") pod \"125ab3c0-c75a-4866-bbfd-d490769e6146\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") "
Nov 02 10:05:44.400752 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400653    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-workload-socket" pod "6f527771-b4d1-43ee-89b4-264cfdf7f4d0" (UID: "6f527771-b4d1-43ee-89b4-264cfdf7f4d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.400752 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400698    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-workload-certs" pod "6f527771-b4d1-43ee-89b4-264cfdf7f4d0" (UID: "6f527771-b4d1-43ee-89b4-264cfdf7f4d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.401220 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400761    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-workload-socket" pod "125ab3c0-c75a-4866-bbfd-d490769e6146" (UID: "125ab3c0-c75a-4866-bbfd-d490769e6146"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.401220 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400861    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-credential-socket" pod "125ab3c0-c75a-4866-bbfd-d490769e6146" (UID: "125ab3c0-c75a-4866-bbfd-d490769e6146"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.401220 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400873    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-data" pod "125ab3c0-c75a-4866-bbfd-d490769e6146" (UID: "125ab3c0-c75a-4866-bbfd-d490769e6146"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.401220 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.400999    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-workload-socket" pod "f0c328ac-256e-4e52-bb7b-81dfac9df62f" (UID: "f0c328ac-256e-4e52-bb7b-81dfac9df62f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.401220 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401000    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-data\") pod \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") "
Nov 02 10:05:44.401220 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401058    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-token\" (UniqueName: \"kubernetes.io/projected/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-token\") pod \"125ab3c0-c75a-4866-bbfd-d490769e6146\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") "
Nov 02 10:05:44.401220 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401094    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-credential-socket\") pod \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") "
Nov 02 10:05:44.401220 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401152    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-envoy\") pod \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") "
Nov 02 10:05:44.401220 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401167    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-data" pod "f0c328ac-256e-4e52-bb7b-81dfac9df62f" (UID: "f0c328ac-256e-4e52-bb7b-81dfac9df62f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.401220 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401186    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c328ac-256e-4e52-bb7b-81dfac9df62f-kube-api-access-x79mx\" (UniqueName: \"kubernetes.io/projected/f0c328ac-256e-4e52-bb7b-81dfac9df62f-kube-api-access-x79mx\") pod \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") "
Nov 02 10:05:44.401220 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401218    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-token\" (UniqueName: \"kubernetes.io/projected/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-token\") pod \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") "
Nov 02 10:05:44.401837 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401252    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/125ab3c0-c75a-4866-bbfd-d490769e6146-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/125ab3c0-c75a-4866-bbfd-d490769e6146-istiod-ca-cert\") pod \"125ab3c0-c75a-4866-bbfd-d490769e6146\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") "
Nov 02 10:05:44.401837 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401282    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istiod-ca-cert\") pod \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") "
Nov 02 10:05:44.401837 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401324    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-token\" (UniqueName: \"kubernetes.io/projected/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-token\") pod \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") "
Nov 02 10:05:44.401837 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401338    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-credential-socket" pod "6f527771-b4d1-43ee-89b4-264cfdf7f4d0" (UID: "6f527771-b4d1-43ee-89b4-264cfdf7f4d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.401837 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401378    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-podinfo\") pod \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") "
Nov 02 10:05:44.401837 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401407    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-workload-certs\") pod \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\" (UID: \"f0c328ac-256e-4e52-bb7b-81dfac9df62f\") "
Nov 02 10:05:44.401837 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401444    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istiod-ca-cert\") pod \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\" (UID: \"6f527771-b4d1-43ee-89b4-264cfdf7f4d0\") "
Nov 02 10:05:44.401837 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401477    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-workload-certs\") pod \"125ab3c0-c75a-4866-bbfd-d490769e6146\" (UID: \"125ab3c0-c75a-4866-bbfd-d490769e6146\") "
Nov 02 10:05:44.401837 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.401808    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-workload-certs" pod "f0c328ac-256e-4e52-bb7b-81dfac9df62f" (UID: "f0c328ac-256e-4e52-bb7b-81dfac9df62f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.402239    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istiod-ca-cert" pod "6f527771-b4d1-43ee-89b4-264cfdf7f4d0" (UID: "6f527771-b4d1-43ee-89b4-264cfdf7f4d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.402580    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125ab3c0-c75a-4866-bbfd-d490769e6146-istiod-ca-cert" pod "125ab3c0-c75a-4866-bbfd-d490769e6146" (UID: "125ab3c0-c75a-4866-bbfd-d490769e6146"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.402929    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-workload-certs" pod "125ab3c0-c75a-4866-bbfd-d490769e6146" (UID: "125ab3c0-c75a-4866-bbfd-d490769e6146"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403172    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403218    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403238    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403253    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403267    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403324    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/125ab3c0-c75a-4866-bbfd-d490769e6146-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403343    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403386    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403400    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403414    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403427    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403461    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403477    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403491    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.403648    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istiod-ca-cert" pod "f0c328ac-256e-4e52-bb7b-81dfac9df62f" (UID: "f0c328ac-256e-4e52-bb7b-81dfac9df62f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.404516    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-podinfo" pod "125ab3c0-c75a-4866-bbfd-d490769e6146" (UID: "125ab3c0-c75a-4866-bbfd-d490769e6146"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.404566    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125ab3c0-c75a-4866-bbfd-d490769e6146-kube-api-access-b9f8g" pod "125ab3c0-c75a-4866-bbfd-d490769e6146" (UID: "125ab3c0-c75a-4866-bbfd-d490769e6146"). InnerVolumeSpecName "kube-api-access-b9f8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.404591    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-podinfo" pod "6f527771-b4d1-43ee-89b4-264cfdf7f4d0" (UID: "6f527771-b4d1-43ee-89b4-264cfdf7f4d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.404763    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-kube-api-access-zzzjg" pod "6f527771-b4d1-43ee-89b4-264cfdf7f4d0" (UID: "6f527771-b4d1-43ee-89b4-264cfdf7f4d0"). InnerVolumeSpecName "kube-api-access-zzzjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.405051    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-envoy" pod "125ab3c0-c75a-4866-bbfd-d490769e6146" (UID: "125ab3c0-c75a-4866-bbfd-d490769e6146"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.405342    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-envoy" pod "6f527771-b4d1-43ee-89b4-264cfdf7f4d0" (UID: "6f527771-b4d1-43ee-89b4-264cfdf7f4d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.405403    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-token" pod "125ab3c0-c75a-4866-bbfd-d490769e6146" (UID: "125ab3c0-c75a-4866-bbfd-d490769e6146"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.405693    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c328ac-256e-4e52-bb7b-81dfac9df62f-kube-api-access-x79mx" pod "f0c328ac-256e-4e52-bb7b-81dfac9df62f" (UID: "f0c328ac-256e-4e52-bb7b-81dfac9df62f"). InnerVolumeSpecName "kube-api-access-x79mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:44.406065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.406034    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-token" pod "f0c328ac-256e-4e52-bb7b-81dfac9df62f" (UID: "f0c328ac-256e-4e52-bb7b-81dfac9df62f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:44.407492 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.406088    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-token" pod "6f527771-b4d1-43ee-89b4-264cfdf7f4d0" (UID: "6f527771-b4d1-43ee-89b4-264cfdf7f4d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:44.407492 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.406108    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-podinfo" pod "f0c328ac-256e-4e52-bb7b-81dfac9df62f" (UID: "f0c328ac-256e-4e52-bb7b-81dfac9df62f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:05:44.407492 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.406208    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-envoy" pod "f0c328ac-256e-4e52-bb7b-81dfac9df62f" (UID: "f0c328ac-256e-4e52-bb7b-81dfac9df62f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:05:44.504491 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.504461    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.504491 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.504487    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.504491 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.504496    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.504706 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.504507    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x79mx\" (UniqueName: \"kubernetes.io/projected/f0c328ac-256e-4e52-bb7b-81dfac9df62f-kube-api-access-x79mx\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.504706 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.504517    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.504706 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.504527    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.504706 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.504535    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.504706 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.504543    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c328ac-256e-4e52-bb7b-81dfac9df62f-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.504706 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.504552    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.504706 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.504560    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.504706 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.504568    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/125ab3c0-c75a-4866-bbfd-d490769e6146-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.504706 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.504576    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzzjg\" (UniqueName: \"kubernetes.io/projected/6f527771-b4d1-43ee-89b4-264cfdf7f4d0-kube-api-access-zzzjg\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:44.504706 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:44.504585    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9f8g\" (UniqueName: \"kubernetes.io/projected/125ab3c0-c75a-4866-bbfd-d490769e6146-kube-api-access-b9f8g\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:45.189658 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.189630    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"
Nov 02 10:05:45.189658 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.189645    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq" event={"ID":"6f527771-b4d1-43ee-89b4-264cfdf7f4d0","Type":"ContainerDied","Data":"94cc95c7d5ac9509ee53e67d63f596657195e29367123c7a53b5c937bc1b72d1"}
Nov 02 10:05:45.190158 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.189709    4614 scope.go:122] "RemoveContainer" containerID="2f087b93cea8c27547aaa5a52530d4a14c04daf23ed726ab7b4a8400b42a1d3e"
Nov 02 10:05:45.195369 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.195317    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd" event={"ID":"125ab3c0-c75a-4866-bbfd-d490769e6146","Type":"ContainerDied","Data":"dd080ac9d37c8f66479b21be6edf3c69bca7dad78c0df2f71f636aac75bae9d7"}
Nov 02 10:05:45.195542 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.195446    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"
Nov 02 10:05:45.200008 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.197779    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m" event={"ID":"f0c328ac-256e-4e52-bb7b-81dfac9df62f","Type":"ContainerDied","Data":"c5576212b8521690393acb53f447e798d4f68b8d528963a8a762672bfdc2bedb"}
Nov 02 10:05:45.200008 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.197808    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"
Nov 02 10:05:45.204016 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.203994    4614 scope.go:122] "RemoveContainer" containerID="619c08182293544fa8052b92d33463817c457f4fe02b637526beee3d12a216e6"
Nov 02 10:05:45.213317 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.213300    4614 scope.go:122] "RemoveContainer" containerID="4f24b79772502bbe789ec2c9a215d33b16fc1fe37d00eab00950d645b7f11372"
Nov 02 10:05:45.220152 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.220128    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"]
Nov 02 10:05:45.225220 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.225200    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-vmkcq"]
Nov 02 10:05:45.239967 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.239939    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"]
Nov 02 10:05:45.247286 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.247262    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-zjvjd"]
Nov 02 10:05:45.267663 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.265533    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"]
Nov 02 10:05:45.268430 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.268403    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-wdc6m"]
Nov 02 10:05:45.315784 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.315743    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" path="/var/lib/kubelet/pods/125ab3c0-c75a-4866-bbfd-d490769e6146/volumes"
Nov 02 10:05:45.316402 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.316382    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" path="/var/lib/kubelet/pods/6f527771-b4d1-43ee-89b4-264cfdf7f4d0/volumes"
Nov 02 10:05:45.316967 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.316948    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" path="/var/lib/kubelet/pods/f0c328ac-256e-4e52-bb7b-81dfac9df62f/volumes"
Nov 02 10:05:45.544946 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.544835    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"]
Nov 02 10:05:45.546981 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:45.546954    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9ae0feb-kh4wf"]
Nov 02 10:05:47.313699 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:47.313667    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bdc4594-88de-4f38-a86b-6c4d3e530932" path="/var/lib/kubelet/pods/7bdc4594-88de-4f38-a86b-6c4d3e530932/volumes"
Nov 02 10:05:52.942291 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:52.942254    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b555b5c85-jv2xg"]
Nov 02 10:05:52.942939 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:52.942468    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5b555b5c85-jv2xg" podUID="43fcb0a2-7557-4805-aff1-ac13b3ba44e4" containerName="manager" containerID="cri-o://04d9524770da0d23925a2dfecc8009167d7580195bd683c1dd84a3a6b0e6a223" gracePeriod=30
Nov 02 10:05:53.194767 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.194701    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 10:05:53.239695 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.239664    4614 generic.go:356] "Generic (PLEG): container finished" podID="43fcb0a2-7557-4805-aff1-ac13b3ba44e4" containerID="04d9524770da0d23925a2dfecc8009167d7580195bd683c1dd84a3a6b0e6a223" exitCode=0
Nov 02 10:05:53.239880 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.239717    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b555b5c85-jv2xg"
Nov 02 10:05:53.239880 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.239752    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b555b5c85-jv2xg" event={"ID":"43fcb0a2-7557-4805-aff1-ac13b3ba44e4","Type":"ContainerDied","Data":"04d9524770da0d23925a2dfecc8009167d7580195bd683c1dd84a3a6b0e6a223"}
Nov 02 10:05:53.239880 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.239812    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b555b5c85-jv2xg" event={"ID":"43fcb0a2-7557-4805-aff1-ac13b3ba44e4","Type":"ContainerDied","Data":"7a3921c42f2d5cd8aebd1bc11e4a78d8ec3f13d98df319ac1174111fa08ee355"}
Nov 02 10:05:53.239880 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.239842    4614 scope.go:122] "RemoveContainer" containerID="04d9524770da0d23925a2dfecc8009167d7580195bd683c1dd84a3a6b0e6a223"
Nov 02 10:05:53.249749 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.249730    4614 scope.go:122] "RemoveContainer" containerID="04d9524770da0d23925a2dfecc8009167d7580195bd683c1dd84a3a6b0e6a223"
Nov 02 10:05:53.250031 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:05:53.250011    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d9524770da0d23925a2dfecc8009167d7580195bd683c1dd84a3a6b0e6a223\": container with ID starting with 04d9524770da0d23925a2dfecc8009167d7580195bd683c1dd84a3a6b0e6a223 not found: ID does not exist" containerID="04d9524770da0d23925a2dfecc8009167d7580195bd683c1dd84a3a6b0e6a223"
Nov 02 10:05:53.250105 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.250043    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d9524770da0d23925a2dfecc8009167d7580195bd683c1dd84a3a6b0e6a223"} err="failed to get container status \"04d9524770da0d23925a2dfecc8009167d7580195bd683c1dd84a3a6b0e6a223\": rpc error: code = NotFound desc = could not find container \"04d9524770da0d23925a2dfecc8009167d7580195bd683c1dd84a3a6b0e6a223\": container with ID starting with 04d9524770da0d23925a2dfecc8009167d7580195bd683c1dd84a3a6b0e6a223 not found: ID does not exist"
Nov 02 10:05:53.379581 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.379549    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-kube-api-access-stf5z\" (UniqueName: \"kubernetes.io/projected/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-kube-api-access-stf5z\") pod \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\" (UID: \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\") "
Nov 02 10:05:53.379786 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.379590    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-webhook-cert\" (UniqueName: \"kubernetes.io/secret/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-webhook-cert\") pod \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\" (UID: \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\") "
Nov 02 10:05:53.379786 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.379642    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-metrics-certs\") pod \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\" (UID: \"43fcb0a2-7557-4805-aff1-ac13b3ba44e4\") "
Nov 02 10:05:53.381729 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.381688    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-kube-api-access-stf5z" pod "43fcb0a2-7557-4805-aff1-ac13b3ba44e4" (UID: "43fcb0a2-7557-4805-aff1-ac13b3ba44e4"). InnerVolumeSpecName "kube-api-access-stf5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:05:53.381729 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.381709    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-metrics-certs" pod "43fcb0a2-7557-4805-aff1-ac13b3ba44e4" (UID: "43fcb0a2-7557-4805-aff1-ac13b3ba44e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:05:53.381880 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.381780    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-webhook-cert" pod "43fcb0a2-7557-4805-aff1-ac13b3ba44e4" (UID: "43fcb0a2-7557-4805-aff1-ac13b3ba44e4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:05:53.480571 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.480544    4614 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-metrics-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:53.480571 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.480571    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stf5z\" (UniqueName: \"kubernetes.io/projected/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-kube-api-access-stf5z\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:53.480756 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.480582    4614 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43fcb0a2-7557-4805-aff1-ac13b3ba44e4-webhook-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:05:53.566547 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.566517    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b555b5c85-jv2xg"]
Nov 02 10:05:53.569398 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:53.569374    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b555b5c85-jv2xg"]
Nov 02 10:05:54.270088 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270053    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b555b5c85-wbwdm"]
Nov 02 10:05:54.270504 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270411    4614 cpu_manager.go:414] "containerMap: removing container" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy"
Nov 02 10:05:54.270504 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270423    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy"
Nov 02 10:05:54.270504 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270445    4614 cpu_manager.go:414] "containerMap: removing container" podUID="43fcb0a2-7557-4805-aff1-ac13b3ba44e4" containerName="manager"
Nov 02 10:05:54.270504 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270450    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43fcb0a2-7557-4805-aff1-ac13b3ba44e4" containerName="manager"
Nov 02 10:05:54.270504 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270461    4614 cpu_manager.go:414] "containerMap: removing container" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy"
Nov 02 10:05:54.270504 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270466    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy"
Nov 02 10:05:54.270504 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270476    4614 cpu_manager.go:414] "containerMap: removing container" podUID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" containerName="istio-proxy"
Nov 02 10:05:54.270504 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270481    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" containerName="istio-proxy"
Nov 02 10:05:54.270504 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270488    4614 cpu_manager.go:414] "containerMap: removing container" podUID="7bdc4594-88de-4f38-a86b-6c4d3e530932" containerName="revoke-keys"
Nov 02 10:05:54.270504 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270492    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bdc4594-88de-4f38-a86b-6c4d3e530932" containerName="revoke-keys"
Nov 02 10:05:54.270854 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270538    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="125ab3c0-c75a-4866-bbfd-d490769e6146" containerName="istio-proxy"
Nov 02 10:05:54.270854 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270547    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f527771-b4d1-43ee-89b4-264cfdf7f4d0" containerName="istio-proxy"
Nov 02 10:05:54.270854 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270552    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bdc4594-88de-4f38-a86b-6c4d3e530932" containerName="revoke-keys"
Nov 02 10:05:54.270854 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270560    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0c328ac-256e-4e52-bb7b-81dfac9df62f" containerName="istio-proxy"
Nov 02 10:05:54.270854 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.270567    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="43fcb0a2-7557-4805-aff1-ac13b3ba44e4" containerName="manager"
Nov 02 10:05:54.274950 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.274931    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:05:54.282336 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.282315    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 10:05:54.287785 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.287758    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b555b5c85-wbwdm"]
Nov 02 10:05:54.292345 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.292320    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 10:05:54.292549 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.292526    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 10:05:54.292669 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.292564    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zrg5v\""
Nov 02 10:05:54.388555 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.388520    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9fed8fc7-4de2-42fe-841d-398f57b7ca33-webhook-cert\") pod \"maas-controller-5b555b5c85-wbwdm\" (UID: \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\") " pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:05:54.388728 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.388633    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmqn9\" (UniqueName: \"kubernetes.io/projected/9fed8fc7-4de2-42fe-841d-398f57b7ca33-kube-api-access-wmqn9\") pod \"maas-controller-5b555b5c85-wbwdm\" (UID: \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\") " pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:05:54.388728 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.388692    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fed8fc7-4de2-42fe-841d-398f57b7ca33-metrics-certs\") pod \"maas-controller-5b555b5c85-wbwdm\" (UID: \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\") " pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:05:54.489028 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.488993    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fed8fc7-4de2-42fe-841d-398f57b7ca33-metrics-certs\") pod \"maas-controller-5b555b5c85-wbwdm\" (UID: \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\") " pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:05:54.489028 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.489028    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9fed8fc7-4de2-42fe-841d-398f57b7ca33-webhook-cert\") pod \"maas-controller-5b555b5c85-wbwdm\" (UID: \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\") " pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:05:54.489278 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.489078    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmqn9\" (UniqueName: \"kubernetes.io/projected/9fed8fc7-4de2-42fe-841d-398f57b7ca33-kube-api-access-wmqn9\") pod \"maas-controller-5b555b5c85-wbwdm\" (UID: \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\") " pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:05:54.491729 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.491701    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9fed8fc7-4de2-42fe-841d-398f57b7ca33-webhook-cert\") pod \"maas-controller-5b555b5c85-wbwdm\" (UID: \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\") " pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:05:54.491855 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.491701    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fed8fc7-4de2-42fe-841d-398f57b7ca33-metrics-certs\") pod \"maas-controller-5b555b5c85-wbwdm\" (UID: \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\") " pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:05:54.502067 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.502042    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmqn9\" (UniqueName: \"kubernetes.io/projected/9fed8fc7-4de2-42fe-841d-398f57b7ca33-kube-api-access-wmqn9\") pod \"maas-controller-5b555b5c85-wbwdm\" (UID: \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\") " pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:05:54.591812 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.591718    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:05:54.883153 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.883129    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b555b5c85-wbwdm"]
Nov 02 10:05:54.885567 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:05:54.885540    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fed8fc7_4de2_42fe_841d_398f57b7ca33.slice/crio-1fa195d35b917dcc309e8cba1407e30d0df3b6faab13baa8cd85d68ebf374b42 WatchSource:0}: Error finding container 1fa195d35b917dcc309e8cba1407e30d0df3b6faab13baa8cd85d68ebf374b42: Status 404 returned error can't find the container with id 1fa195d35b917dcc309e8cba1407e30d0df3b6faab13baa8cd85d68ebf374b42
Nov 02 10:05:54.886911 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:54.886891    4614 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 10:05:55.253597 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:55.253553    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b555b5c85-wbwdm" event={"ID":"9fed8fc7-4de2-42fe-841d-398f57b7ca33","Type":"ContainerStarted","Data":"1fa195d35b917dcc309e8cba1407e30d0df3b6faab13baa8cd85d68ebf374b42"}
Nov 02 10:05:55.313943 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:55.313908    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43fcb0a2-7557-4805-aff1-ac13b3ba44e4" path="/var/lib/kubelet/pods/43fcb0a2-7557-4805-aff1-ac13b3ba44e4/volumes"
Nov 02 10:05:56.261262 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:56.261220    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b555b5c85-wbwdm" event={"ID":"9fed8fc7-4de2-42fe-841d-398f57b7ca33","Type":"ContainerStarted","Data":"d779196c723830ee665295905568b7c167858c9ec1351a05ddf9c235f19f42b3"}
Nov 02 10:05:56.261503 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:56.261337    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:05:56.283555 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:05:56.283488    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5b555b5c85-wbwdm" podStartSLOduration=1.674990567 podStartE2EDuration="2.283469338s" totalImagesPullingTime="608.478771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:05:56.282435191 +0000 UTC m=+1837.115093273" watchObservedRunningTime="2026-11-02 10:05:56.283469338 +0000 UTC m=+1837.116127419"
Nov 02 10:06:07.267934 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:07.267900    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:06:09.033398 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.033360    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"]
Nov 02 10:06:09.036694 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.036673    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.053190 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.053147    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"]
Nov 02 10:06:09.054881 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.054864    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"]
Nov 02 10:06:09.058144 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.058122    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"]
Nov 02 10:06:09.058263 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.058245    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.061197 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.061181    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.068893 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.068868    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"]
Nov 02 10:06:09.091760 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.091731    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"]
Nov 02 10:06:09.116262 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116236    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.116394 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116269    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.116394 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116290    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.116394 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116337    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn6gf\" (UniqueName: \"kubernetes.io/projected/14493b0f-4cb8-474a-ae5d-216823f6d2be-kube-api-access-jn6gf\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.116394 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116373    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcscw\" (UniqueName: \"kubernetes.io/projected/67fe0167-799a-4caa-9318-ef604f0ca331-kube-api-access-dcscw\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.116539 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116444    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.116589 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116573    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67fe0167-799a-4caa-9318-ef604f0ca331-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.116629 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116600    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.116669 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116627    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.116703 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116673    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67fe0167-799a-4caa-9318-ef604f0ca331-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.116703 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116692    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.116767 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116748    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.116801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116769    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.116801 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116789    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14493b0f-4cb8-474a-ae5d-216823f6d2be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.116870 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116805    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.116904 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116878    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.116948 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116935    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.116985 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.116955    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkj7\" (UniqueName: \"kubernetes.io/projected/69ff7ab2-6bde-4cad-8459-8fad2688572b-kube-api-access-dbkj7\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.117020 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.117005    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.117054 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.117042    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.117089 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.117076    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.117122 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.117095    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ff7ab2-6bde-4cad-8459-8fad2688572b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.117122 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.117114    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.117200 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.117139    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.117200 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.117176    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.117200 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.117193    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.117322 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.117265    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67fe0167-799a-4caa-9318-ef604f0ca331-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.218114 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218085    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.218290 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218123    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.218290 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218141    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkj7\" (UniqueName: \"kubernetes.io/projected/69ff7ab2-6bde-4cad-8459-8fad2688572b-kube-api-access-dbkj7\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.218290 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218167    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.218290 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218192    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.218290 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218214    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.218290 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218242    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ff7ab2-6bde-4cad-8459-8fad2688572b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.218290 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218268    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.218751 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218324    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.218751 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218386    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.218751 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218414    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.218751 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218450    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67fe0167-799a-4caa-9318-ef604f0ca331-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.218751 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218486    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.218751 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218520    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.218751 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218553    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.218751 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218579    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn6gf\" (UniqueName: \"kubernetes.io/projected/14493b0f-4cb8-474a-ae5d-216823f6d2be-kube-api-access-jn6gf\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.218751 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218595    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.218751 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218660    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.218751 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218700    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.219416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218843    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.219416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218932    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.219416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219026    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.219416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219064    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ff7ab2-6bde-4cad-8459-8fad2688572b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.219416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.218603    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcscw\" (UniqueName: \"kubernetes.io/projected/67fe0167-799a-4caa-9318-ef604f0ca331-kube-api-access-dcscw\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.219416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219120    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.219416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219160    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67fe0167-799a-4caa-9318-ef604f0ca331-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.219416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219191    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.219416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219217    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.219416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219255    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67fe0167-799a-4caa-9318-ef604f0ca331-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.219416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219293    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.219416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219321    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.219416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219323    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.219416 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219415    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.220184 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219438    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67fe0167-799a-4caa-9318-ef604f0ca331-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.220184 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219449    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14493b0f-4cb8-474a-ae5d-216823f6d2be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.220184 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219496    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.220184 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219543    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.220184 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219781    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.220184 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.219801    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.220184 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.220047    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14493b0f-4cb8-474a-ae5d-216823f6d2be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.220184 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.220049    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.220506 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.220211    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.221074 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.221041    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.221782 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.221751    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.221894 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.221751    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.221894 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.221841    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.222677 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.222658    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.222759 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.222674    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67fe0167-799a-4caa-9318-ef604f0ca331-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.230147 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.230124    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.230576 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.230559    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcscw\" (UniqueName: \"kubernetes.io/projected/67fe0167-799a-4caa-9318-ef604f0ca331-kube-api-access-dcscw\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.230992 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.230975    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkj7\" (UniqueName: \"kubernetes.io/projected/69ff7ab2-6bde-4cad-8459-8fad2688572b-kube-api-access-dbkj7\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2lqv8\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.231906 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.231870    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn6gf\" (UniqueName: \"kubernetes.io/projected/14493b0f-4cb8-474a-ae5d-216823f6d2be-kube-api-access-jn6gf\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.233331 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.233314    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-4rbgx\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.233651 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.233634    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67fe0167-799a-4caa-9318-ef604f0ca331-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5msqr\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.349367 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.349269    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:09.375439 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.375409    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:09.381532 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.381507    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:09.631866 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.631842    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"]
Nov 02 10:06:09.634863 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:06:09.634828    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fe0167_799a_4caa_9318_ef604f0ca331.slice/crio-803afbc5dc0789694d68e861ec5e6cafa2f7eaa8ad6f6c58102a5a56ed288712 WatchSource:0}: Error finding container 803afbc5dc0789694d68e861ec5e6cafa2f7eaa8ad6f6c58102a5a56ed288712: Status 404 returned error can't find the container with id 803afbc5dc0789694d68e861ec5e6cafa2f7eaa8ad6f6c58102a5a56ed288712
Nov 02 10:06:09.636952 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.636921    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:06:09.637044 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.636987    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:06:09.637044 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.637019    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:06:09.637459 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:06:09.637374    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:06:09.696550 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.696526    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"]
Nov 02 10:06:09.698863 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:06:09.698840    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14493b0f_4cb8_474a_ae5d_216823f6d2be.slice/crio-3a3a2c9d6ae1f29bc32917ecd1482be3555e04f9ebdf5b037587ef8169dc97c7 WatchSource:0}: Error finding container 3a3a2c9d6ae1f29bc32917ecd1482be3555e04f9ebdf5b037587ef8169dc97c7: Status 404 returned error can't find the container with id 3a3a2c9d6ae1f29bc32917ecd1482be3555e04f9ebdf5b037587ef8169dc97c7
Nov 02 10:06:09.701181 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.701148    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:06:09.701268 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.701217    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:06:09.701268 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.701248    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:06:09.708395 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.708374    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"]
Nov 02 10:06:09.710622 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:06:09.710594    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ff7ab2_6bde_4cad_8459_8fad2688572b.slice/crio-91db7f4cdf22015bf3c402f72b0829f28745f7461e763f6a9295b7a0682864f8 WatchSource:0}: Error finding container 91db7f4cdf22015bf3c402f72b0829f28745f7461e763f6a9295b7a0682864f8: Status 404 returned error can't find the container with id 91db7f4cdf22015bf3c402f72b0829f28745f7461e763f6a9295b7a0682864f8
Nov 02 10:06:09.713808 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.713772    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:06:09.713878 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.713853    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:06:09.713924 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:09.713900    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:06:10.023967 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.023933    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"]
Nov 02 10:06:10.031177 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.031147    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e904c88-c6vbt"]
Nov 02 10:06:10.345434 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.339130    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" event={"ID":"69ff7ab2-6bde-4cad-8459-8fad2688572b","Type":"ContainerStarted","Data":"990910fd6fe246a46f0dd8124ad3fb0e82cc2e5d6397e1625ec878381dd9b4a3"}
Nov 02 10:06:10.345434 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.339201    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" event={"ID":"69ff7ab2-6bde-4cad-8459-8fad2688572b","Type":"ContainerStarted","Data":"91db7f4cdf22015bf3c402f72b0829f28745f7461e763f6a9295b7a0682864f8"}
Nov 02 10:06:10.355376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.353812    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" event={"ID":"67fe0167-799a-4caa-9318-ef604f0ca331","Type":"ContainerStarted","Data":"39d45fa80450268c3ffbc1333293fb1c27afc5b5922a27455028b6ed1d2ae256"}
Nov 02 10:06:10.355376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.353874    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" event={"ID":"67fe0167-799a-4caa-9318-ef604f0ca331","Type":"ContainerStarted","Data":"803afbc5dc0789694d68e861ec5e6cafa2f7eaa8ad6f6c58102a5a56ed288712"}
Nov 02 10:06:10.372386 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.372022    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" event={"ID":"14493b0f-4cb8-474a-ae5d-216823f6d2be","Type":"ContainerStarted","Data":"24d9804661f73184e1c2ebe886e02c653a50120f2b1f6a64e01658cc1f9aa1c9"}
Nov 02 10:06:10.372386 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.372086    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" event={"ID":"14493b0f-4cb8-474a-ae5d-216823f6d2be","Type":"ContainerStarted","Data":"3a3a2c9d6ae1f29bc32917ecd1482be3555e04f9ebdf5b037587ef8169dc97c7"}
Nov 02 10:06:10.378559 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.378102    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:10.378559 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.378232    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 10:06:10.378559 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.378279    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 10:06:10.387045 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.386959    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:10.388829 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.388801    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 10:06:10.389060 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.389035    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 10:06:10.393387 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.391204    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podStartSLOduration=1.391188225 podStartE2EDuration="1.391188225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:06:10.389490267 +0000 UTC m=+1851.222148349" watchObservedRunningTime="2026-11-02 10:06:10.391188225 +0000 UTC m=+1851.223846305"
Nov 02 10:06:10.416985 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.416922    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podStartSLOduration=1.416908442 podStartE2EDuration="1.416908442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:06:10.415434884 +0000 UTC m=+1851.248092958" watchObservedRunningTime="2026-11-02 10:06:10.416908442 +0000 UTC m=+1851.249566522"
Nov 02 10:06:10.497377 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:10.481034    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podStartSLOduration=1.48101283 podStartE2EDuration="1.48101283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:06:10.465959194 +0000 UTC m=+1851.298617287" watchObservedRunningTime="2026-11-02 10:06:10.48101283 +0000 UTC m=+1851.313670913"
Nov 02 10:06:11.313997 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:11.313963    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="493137e8-645a-4e4e-a995-9a6f29d8d645" path="/var/lib/kubelet/pods/493137e8-645a-4e4e-a995-9a6f29d8d645/volumes"
Nov 02 10:06:11.351940 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:11.349704    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:11.356372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:11.353453    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 10:06:11.356372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:11.353508    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 10:06:11.379374 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:11.378511    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 10:06:11.379374 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:11.378566    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 10:06:11.387373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:11.386468    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 10:06:11.387373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:11.386577    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 10:06:12.353417 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:12.353378    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 10:06:12.354035 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:12.354005    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 10:06:12.378848 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:12.378813    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 10:06:12.379208 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:12.379164    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 10:06:12.383747 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:12.383708    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 10:06:12.383938 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:12.383916    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 10:06:13.269375 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.262438    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"]
Nov 02 10:06:13.290376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.274620    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"]
Nov 02 10:06:13.290376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.274745    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"
Nov 02 10:06:13.311856 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.291887    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:06:13.311856 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.291933    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:06:13.349808 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.349771    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 10:06:13.350077 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.350046    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 10:06:13.378296 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.378261    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 10:06:13.388395 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.383913    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 10:06:13.407553 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.388925    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 10:06:13.407553 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.400459    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 10:06:13.417243 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.417217    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55107c99-4d99-4ac4-83eb-b9563f43aea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-t8nv6\" (UID: \"55107c99-4d99-4ac4-83eb-b9563f43aea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"
Nov 02 10:06:13.417460 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.417283    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5gzq\" (UniqueName: \"kubernetes.io/projected/55107c99-4d99-4ac4-83eb-b9563f43aea1-kube-api-access-g5gzq\") pod \"maas-api-revoke-sub-85237e45-t8nv6\" (UID: \"55107c99-4d99-4ac4-83eb-b9563f43aea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"
Nov 02 10:06:13.417460 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.417398    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55107c99-4d99-4ac4-83eb-b9563f43aea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-t8nv6\" (UID: \"55107c99-4d99-4ac4-83eb-b9563f43aea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"
Nov 02 10:06:13.518862 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.518810    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5gzq\" (UniqueName: \"kubernetes.io/projected/55107c99-4d99-4ac4-83eb-b9563f43aea1-kube-api-access-g5gzq\") pod \"maas-api-revoke-sub-85237e45-t8nv6\" (UID: \"55107c99-4d99-4ac4-83eb-b9563f43aea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"
Nov 02 10:06:13.519030 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.518909    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55107c99-4d99-4ac4-83eb-b9563f43aea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-t8nv6\" (UID: \"55107c99-4d99-4ac4-83eb-b9563f43aea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"
Nov 02 10:06:13.519074 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.519025    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55107c99-4d99-4ac4-83eb-b9563f43aea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-t8nv6\" (UID: \"55107c99-4d99-4ac4-83eb-b9563f43aea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"
Nov 02 10:06:13.519788 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.519726    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55107c99-4d99-4ac4-83eb-b9563f43aea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-t8nv6\" (UID: \"55107c99-4d99-4ac4-83eb-b9563f43aea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"
Nov 02 10:06:13.521565 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.521545    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55107c99-4d99-4ac4-83eb-b9563f43aea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-t8nv6\" (UID: \"55107c99-4d99-4ac4-83eb-b9563f43aea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"
Nov 02 10:06:13.558157 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.558088    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5gzq\" (UniqueName: \"kubernetes.io/projected/55107c99-4d99-4ac4-83eb-b9563f43aea1-kube-api-access-g5gzq\") pod \"maas-api-revoke-sub-85237e45-t8nv6\" (UID: \"55107c99-4d99-4ac4-83eb-b9563f43aea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"
Nov 02 10:06:13.619912 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:13.619879    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"
Nov 02 10:06:14.108809 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:14.101186    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"]
Nov 02 10:06:14.109884 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:06:14.109855    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55107c99_4d99_4ac4_83eb_b9563f43aea1.slice/crio-e448a0e8c2e2c950be06354252bb0ccfca1432f6c17c1f95a7c3efd121cb47b2 WatchSource:0}: Error finding container e448a0e8c2e2c950be06354252bb0ccfca1432f6c17c1f95a7c3efd121cb47b2: Status 404 returned error can't find the container with id e448a0e8c2e2c950be06354252bb0ccfca1432f6c17c1f95a7c3efd121cb47b2
Nov 02 10:06:14.353208 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:14.352739    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 10:06:14.353208 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:14.352803    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 10:06:14.382430 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:14.380881    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 10:06:14.382430 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:14.380940    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 10:06:14.382430 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:14.382028    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 10:06:14.382430 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:14.382080    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 10:06:14.420757 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:14.420719    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6" event={"ID":"55107c99-4d99-4ac4-83eb-b9563f43aea1","Type":"ContainerStarted","Data":"e448a0e8c2e2c950be06354252bb0ccfca1432f6c17c1f95a7c3efd121cb47b2"}
Nov 02 10:06:15.351245 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:15.350767    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 10:06:15.351245 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:15.350829    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 10:06:15.376687 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:15.376623    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 10:06:15.377062 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:15.376966    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 10:06:15.386327 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:15.386294    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 10:06:15.387079 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:15.387006    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 10:06:15.432154 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:15.431629    4614 generic.go:356] "Generic (PLEG): container finished" podID="55107c99-4d99-4ac4-83eb-b9563f43aea1" containerID="dfb8be980a9eebacde607d232eb1677174506c7bf88173ed4efbd30e543bb3ef" exitCode=0
Nov 02 10:06:15.432154 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:15.431697    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6" event={"ID":"55107c99-4d99-4ac4-83eb-b9563f43aea1","Type":"ContainerDied","Data":"dfb8be980a9eebacde607d232eb1677174506c7bf88173ed4efbd30e543bb3ef"}
Nov 02 10:06:16.350136 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.350093    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 10:06:16.350326 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.350160    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 10:06:16.376619 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.376578    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 10:06:16.376741 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.376637    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 10:06:16.381887 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.381862    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 10:06:16.381965 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.381913    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 10:06:16.593399 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.593365    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"
Nov 02 10:06:16.690135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.690040    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55107c99-4d99-4ac4-83eb-b9563f43aea1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55107c99-4d99-4ac4-83eb-b9563f43aea1-maas-api-xks-ca\") pod \"55107c99-4d99-4ac4-83eb-b9563f43aea1\" (UID: \"55107c99-4d99-4ac4-83eb-b9563f43aea1\") "
Nov 02 10:06:16.690135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.690107    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55107c99-4d99-4ac4-83eb-b9563f43aea1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55107c99-4d99-4ac4-83eb-b9563f43aea1-maas-api-openshift-ca\") pod \"55107c99-4d99-4ac4-83eb-b9563f43aea1\" (UID: \"55107c99-4d99-4ac4-83eb-b9563f43aea1\") "
Nov 02 10:06:16.690397 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.690213    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55107c99-4d99-4ac4-83eb-b9563f43aea1-kube-api-access-g5gzq\" (UniqueName: \"kubernetes.io/projected/55107c99-4d99-4ac4-83eb-b9563f43aea1-kube-api-access-g5gzq\") pod \"55107c99-4d99-4ac4-83eb-b9563f43aea1\" (UID: \"55107c99-4d99-4ac4-83eb-b9563f43aea1\") "
Nov 02 10:06:16.690678 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.690642    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55107c99-4d99-4ac4-83eb-b9563f43aea1-maas-api-openshift-ca" pod "55107c99-4d99-4ac4-83eb-b9563f43aea1" (UID: "55107c99-4d99-4ac4-83eb-b9563f43aea1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:06:16.692618 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.692536    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55107c99-4d99-4ac4-83eb-b9563f43aea1-maas-api-xks-ca" pod "55107c99-4d99-4ac4-83eb-b9563f43aea1" (UID: "55107c99-4d99-4ac4-83eb-b9563f43aea1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:06:16.693026 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.692997    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55107c99-4d99-4ac4-83eb-b9563f43aea1-kube-api-access-g5gzq" pod "55107c99-4d99-4ac4-83eb-b9563f43aea1" (UID: "55107c99-4d99-4ac4-83eb-b9563f43aea1"). InnerVolumeSpecName "kube-api-access-g5gzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:06:16.791939 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.791893    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55107c99-4d99-4ac4-83eb-b9563f43aea1-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:06:16.791939 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.791935    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55107c99-4d99-4ac4-83eb-b9563f43aea1-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:06:16.792192 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:16.791952    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5gzq\" (UniqueName: \"kubernetes.io/projected/55107c99-4d99-4ac4-83eb-b9563f43aea1-kube-api-access-g5gzq\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:06:17.349978 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:17.349945    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 02 10:06:17.350128 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:17.350008    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 02 10:06:17.376712 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:17.376681    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 02 10:06:17.376827 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:17.376736    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 02 10:06:17.382122 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:17.382098    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 02 10:06:17.382203 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:17.382145    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 02 10:06:17.446763 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:17.446724    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6" event={"ID":"55107c99-4d99-4ac4-83eb-b9563f43aea1","Type":"ContainerDied","Data":"e448a0e8c2e2c950be06354252bb0ccfca1432f6c17c1f95a7c3efd121cb47b2"}
Nov 02 10:06:17.446924 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:17.446765    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"
Nov 02 10:06:17.446924 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:17.446787    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e448a0e8c2e2c950be06354252bb0ccfca1432f6c17c1f95a7c3efd121cb47b2"
Nov 02 10:06:19.350781 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:19.350744    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:06:19.351275 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:19.350806    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:06:19.416535 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:19.416489    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:06:19.416723 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:19.416575    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:06:19.423499 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:19.423466    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:06:19.423642 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:19.423516    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:06:20.392569 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:20.392516    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:06:20.392970 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:20.392594    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:06:20.417578 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:20.417534    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:06:20.417764 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:20.417615    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:06:20.424252 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:20.424220    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:06:20.424408 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:20.424276    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:06:21.440540 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:21.440486    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:06:21.441138 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:21.441086    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:06:21.468473 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:21.468436    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:06:21.468694 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:21.468663    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:06:21.469545 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:21.469521    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:06:21.469694 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:21.469673    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:06:22.023183 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:22.023147    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"]
Nov 02 10:06:22.029383 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:22.026219    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f937c0-pfrkt"]
Nov 02 10:06:22.457433 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:22.456509    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:06:22.457433 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:22.456597    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:06:22.512747 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:22.512476    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:06:22.512747 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:22.512488    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:06:22.512747 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:22.512519    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:06:22.512747 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:22.512540    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:06:22.548441 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:22.548409    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:22.549578 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:22.549509    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:22.550505 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:22.550476    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:06:23.231003 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:23.230966    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:23.231494 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:23.231469    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:23.316974 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:23.316943    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ae2c6f-4236-4d28-8c81-94de9c192785" path="/var/lib/kubelet/pods/03ae2c6f-4236-4d28-8c81-94de9c192785/volumes"
Nov 02 10:06:23.374311 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:23.374281    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:23.374620 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:23.374599    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:23.375260 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:23.375243    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:06:23.403311 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:23.403289    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:06:24.857189 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:24.857151    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b555b5c85-wbwdm"]
Nov 02 10:06:24.857763 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:24.857374    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5b555b5c85-wbwdm" podUID="9fed8fc7-4de2-42fe-841d-398f57b7ca33" containerName="manager" containerID="cri-o://d779196c723830ee665295905568b7c167858c9ec1351a05ddf9c235f19f42b3" gracePeriod=30
Nov 02 10:06:25.102590 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.102565    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:06:25.201101 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.201020    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fed8fc7-4de2-42fe-841d-398f57b7ca33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fed8fc7-4de2-42fe-841d-398f57b7ca33-metrics-certs\") pod \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\" (UID: \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\") "
Nov 02 10:06:25.201101 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.201058    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fed8fc7-4de2-42fe-841d-398f57b7ca33-kube-api-access-wmqn9\" (UniqueName: \"kubernetes.io/projected/9fed8fc7-4de2-42fe-841d-398f57b7ca33-kube-api-access-wmqn9\") pod \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\" (UID: \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\") "
Nov 02 10:06:25.201101 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.201096    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fed8fc7-4de2-42fe-841d-398f57b7ca33-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9fed8fc7-4de2-42fe-841d-398f57b7ca33-webhook-cert\") pod \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\" (UID: \"9fed8fc7-4de2-42fe-841d-398f57b7ca33\") "
Nov 02 10:06:25.203023 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.202991    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fed8fc7-4de2-42fe-841d-398f57b7ca33-metrics-certs" pod "9fed8fc7-4de2-42fe-841d-398f57b7ca33" (UID: "9fed8fc7-4de2-42fe-841d-398f57b7ca33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:06:25.203153 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.203133    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fed8fc7-4de2-42fe-841d-398f57b7ca33-kube-api-access-wmqn9" pod "9fed8fc7-4de2-42fe-841d-398f57b7ca33" (UID: "9fed8fc7-4de2-42fe-841d-398f57b7ca33"). InnerVolumeSpecName "kube-api-access-wmqn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:06:25.203218 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.203198    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fed8fc7-4de2-42fe-841d-398f57b7ca33-webhook-cert" pod "9fed8fc7-4de2-42fe-841d-398f57b7ca33" (UID: "9fed8fc7-4de2-42fe-841d-398f57b7ca33"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:06:25.302604 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.302568    4614 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9fed8fc7-4de2-42fe-841d-398f57b7ca33-webhook-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:06:25.302604 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.302597    4614 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fed8fc7-4de2-42fe-841d-398f57b7ca33-metrics-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:06:25.302604 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.302608    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmqn9\" (UniqueName: \"kubernetes.io/projected/9fed8fc7-4de2-42fe-841d-398f57b7ca33-kube-api-access-wmqn9\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:06:25.519616 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.519585    4614 generic.go:356] "Generic (PLEG): container finished" podID="9fed8fc7-4de2-42fe-841d-398f57b7ca33" containerID="d779196c723830ee665295905568b7c167858c9ec1351a05ddf9c235f19f42b3" exitCode=0
Nov 02 10:06:25.519815 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.519631    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b555b5c85-wbwdm"
Nov 02 10:06:25.519815 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.519638    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b555b5c85-wbwdm" event={"ID":"9fed8fc7-4de2-42fe-841d-398f57b7ca33","Type":"ContainerDied","Data":"d779196c723830ee665295905568b7c167858c9ec1351a05ddf9c235f19f42b3"}
Nov 02 10:06:25.519815 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.519679    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b555b5c85-wbwdm" event={"ID":"9fed8fc7-4de2-42fe-841d-398f57b7ca33","Type":"ContainerDied","Data":"1fa195d35b917dcc309e8cba1407e30d0df3b6faab13baa8cd85d68ebf374b42"}
Nov 02 10:06:25.519815 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.519699    4614 scope.go:122] "RemoveContainer" containerID="d779196c723830ee665295905568b7c167858c9ec1351a05ddf9c235f19f42b3"
Nov 02 10:06:25.528378 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.528344    4614 scope.go:122] "RemoveContainer" containerID="d779196c723830ee665295905568b7c167858c9ec1351a05ddf9c235f19f42b3"
Nov 02 10:06:25.528674 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:06:25.528654    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d779196c723830ee665295905568b7c167858c9ec1351a05ddf9c235f19f42b3\": container with ID starting with d779196c723830ee665295905568b7c167858c9ec1351a05ddf9c235f19f42b3 not found: ID does not exist" containerID="d779196c723830ee665295905568b7c167858c9ec1351a05ddf9c235f19f42b3"
Nov 02 10:06:25.528732 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.528683    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d779196c723830ee665295905568b7c167858c9ec1351a05ddf9c235f19f42b3"} err="failed to get container status \"d779196c723830ee665295905568b7c167858c9ec1351a05ddf9c235f19f42b3\": rpc error: code = NotFound desc = could not find container \"d779196c723830ee665295905568b7c167858c9ec1351a05ddf9c235f19f42b3\": container with ID starting with d779196c723830ee665295905568b7c167858c9ec1351a05ddf9c235f19f42b3 not found: ID does not exist"
Nov 02 10:06:25.538271 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.538244    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b555b5c85-wbwdm"]
Nov 02 10:06:25.544165 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:25.544141    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b555b5c85-wbwdm"]
Nov 02 10:06:27.314670 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:27.314638    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fed8fc7-4de2-42fe-841d-398f57b7ca33" path="/var/lib/kubelet/pods/9fed8fc7-4de2-42fe-841d-398f57b7ca33/volumes"
Nov 02 10:06:39.634745 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:06:39.634709    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:06:52.869284 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:52.869246    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"]
Nov 02 10:06:52.869699 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:52.869613    4614 cpu_manager.go:414] "containerMap: removing container" podUID="55107c99-4d99-4ac4-83eb-b9563f43aea1" containerName="revoke-keys"
Nov 02 10:06:52.869699 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:52.869625    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55107c99-4d99-4ac4-83eb-b9563f43aea1" containerName="revoke-keys"
Nov 02 10:06:52.869699 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:52.869635    4614 cpu_manager.go:414] "containerMap: removing container" podUID="9fed8fc7-4de2-42fe-841d-398f57b7ca33" containerName="manager"
Nov 02 10:06:52.869699 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:52.869640    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fed8fc7-4de2-42fe-841d-398f57b7ca33" containerName="manager"
Nov 02 10:06:52.869699 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:52.869699    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="55107c99-4d99-4ac4-83eb-b9563f43aea1" containerName="revoke-keys"
Nov 02 10:06:52.869869 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:52.869706    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fed8fc7-4de2-42fe-841d-398f57b7ca33" containerName="manager"
Nov 02 10:06:52.872904 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:52.872888    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"
Nov 02 10:06:52.880383 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:52.879873    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:06:52.880383 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:52.880102    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:06:52.882682 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:52.882658    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"]
Nov 02 10:06:52.980998 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:52.980966    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc668fb-131c-424c-94e1-9c56c8a726dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-c9kwh\" (UID: \"7cc668fb-131c-424c-94e1-9c56c8a726dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"
Nov 02 10:06:52.981155 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:52.981016    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7zsv\" (UniqueName: \"kubernetes.io/projected/7cc668fb-131c-424c-94e1-9c56c8a726dd-kube-api-access-p7zsv\") pod \"maas-api-revoke-sub-f904bff9-c9kwh\" (UID: \"7cc668fb-131c-424c-94e1-9c56c8a726dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"
Nov 02 10:06:52.981155 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:52.981113    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc668fb-131c-424c-94e1-9c56c8a726dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-c9kwh\" (UID: \"7cc668fb-131c-424c-94e1-9c56c8a726dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"
Nov 02 10:06:53.082292 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:53.082257    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc668fb-131c-424c-94e1-9c56c8a726dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-c9kwh\" (UID: \"7cc668fb-131c-424c-94e1-9c56c8a726dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"
Nov 02 10:06:53.082497 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:53.082310    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7zsv\" (UniqueName: \"kubernetes.io/projected/7cc668fb-131c-424c-94e1-9c56c8a726dd-kube-api-access-p7zsv\") pod \"maas-api-revoke-sub-f904bff9-c9kwh\" (UID: \"7cc668fb-131c-424c-94e1-9c56c8a726dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"
Nov 02 10:06:53.082497 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:53.082373    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc668fb-131c-424c-94e1-9c56c8a726dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-c9kwh\" (UID: \"7cc668fb-131c-424c-94e1-9c56c8a726dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"
Nov 02 10:06:53.082943 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:53.082915    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc668fb-131c-424c-94e1-9c56c8a726dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-c9kwh\" (UID: \"7cc668fb-131c-424c-94e1-9c56c8a726dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"
Nov 02 10:06:53.084662 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:53.084641    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc668fb-131c-424c-94e1-9c56c8a726dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-c9kwh\" (UID: \"7cc668fb-131c-424c-94e1-9c56c8a726dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"
Nov 02 10:06:53.094191 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:53.094170    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7zsv\" (UniqueName: \"kubernetes.io/projected/7cc668fb-131c-424c-94e1-9c56c8a726dd-kube-api-access-p7zsv\") pod \"maas-api-revoke-sub-f904bff9-c9kwh\" (UID: \"7cc668fb-131c-424c-94e1-9c56c8a726dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"
Nov 02 10:06:53.187080 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:53.187001    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"
Nov 02 10:06:53.460116 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:53.459915    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"]
Nov 02 10:06:53.462564 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:06:53.462538    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc668fb_131c_424c_94e1_9c56c8a726dd.slice/crio-c5944b4eb9cf5d5da9ea8522900d5f0233c4015a8dd3af61b34ba295308bfe43 WatchSource:0}: Error finding container c5944b4eb9cf5d5da9ea8522900d5f0233c4015a8dd3af61b34ba295308bfe43: Status 404 returned error can't find the container with id c5944b4eb9cf5d5da9ea8522900d5f0233c4015a8dd3af61b34ba295308bfe43
Nov 02 10:06:53.670969 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:53.670936    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh" event={"ID":"7cc668fb-131c-424c-94e1-9c56c8a726dd","Type":"ContainerStarted","Data":"c5944b4eb9cf5d5da9ea8522900d5f0233c4015a8dd3af61b34ba295308bfe43"}
Nov 02 10:06:54.677309 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:54.677196    4614 generic.go:356] "Generic (PLEG): container finished" podID="7cc668fb-131c-424c-94e1-9c56c8a726dd" containerID="ccd0ad18d049ec6d892e19e158d52f45555af7c54a669e8adea8683aaf160640" exitCode=0
Nov 02 10:06:54.677309 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:54.677278    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh" event={"ID":"7cc668fb-131c-424c-94e1-9c56c8a726dd","Type":"ContainerDied","Data":"ccd0ad18d049ec6d892e19e158d52f45555af7c54a669e8adea8683aaf160640"}
Nov 02 10:06:55.810735 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:55.810708    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"
Nov 02 10:06:55.907862 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:55.907826    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cc668fb-131c-424c-94e1-9c56c8a726dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc668fb-131c-424c-94e1-9c56c8a726dd-maas-api-openshift-ca\") pod \"7cc668fb-131c-424c-94e1-9c56c8a726dd\" (UID: \"7cc668fb-131c-424c-94e1-9c56c8a726dd\") "
Nov 02 10:06:55.908159 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:55.907895    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc668fb-131c-424c-94e1-9c56c8a726dd-kube-api-access-p7zsv\" (UniqueName: \"kubernetes.io/projected/7cc668fb-131c-424c-94e1-9c56c8a726dd-kube-api-access-p7zsv\") pod \"7cc668fb-131c-424c-94e1-9c56c8a726dd\" (UID: \"7cc668fb-131c-424c-94e1-9c56c8a726dd\") "
Nov 02 10:06:55.908159 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:55.907987    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cc668fb-131c-424c-94e1-9c56c8a726dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc668fb-131c-424c-94e1-9c56c8a726dd-maas-api-xks-ca\") pod \"7cc668fb-131c-424c-94e1-9c56c8a726dd\" (UID: \"7cc668fb-131c-424c-94e1-9c56c8a726dd\") "
Nov 02 10:06:55.908276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:55.908194    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc668fb-131c-424c-94e1-9c56c8a726dd-maas-api-openshift-ca" pod "7cc668fb-131c-424c-94e1-9c56c8a726dd" (UID: "7cc668fb-131c-424c-94e1-9c56c8a726dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:06:55.910063 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:55.910038    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc668fb-131c-424c-94e1-9c56c8a726dd-maas-api-xks-ca" pod "7cc668fb-131c-424c-94e1-9c56c8a726dd" (UID: "7cc668fb-131c-424c-94e1-9c56c8a726dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:06:55.910128 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:55.910042    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc668fb-131c-424c-94e1-9c56c8a726dd-kube-api-access-p7zsv" pod "7cc668fb-131c-424c-94e1-9c56c8a726dd" (UID: "7cc668fb-131c-424c-94e1-9c56c8a726dd"). InnerVolumeSpecName "kube-api-access-p7zsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:06:56.008788 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:56.008757    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7zsv\" (UniqueName: \"kubernetes.io/projected/7cc668fb-131c-424c-94e1-9c56c8a726dd-kube-api-access-p7zsv\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:06:56.008788 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:56.008784    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cc668fb-131c-424c-94e1-9c56c8a726dd-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:06:56.008788 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:56.008795    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cc668fb-131c-424c-94e1-9c56c8a726dd-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:06:56.689184 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:56.689153    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"
Nov 02 10:06:56.689184 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:56.689180    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh" event={"ID":"7cc668fb-131c-424c-94e1-9c56c8a726dd","Type":"ContainerDied","Data":"c5944b4eb9cf5d5da9ea8522900d5f0233c4015a8dd3af61b34ba295308bfe43"}
Nov 02 10:06:56.689486 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:06:56.689221    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5944b4eb9cf5d5da9ea8522900d5f0233c4015a8dd3af61b34ba295308bfe43"
Nov 02 10:07:08.604373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:07:08.603499    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:07:08.604373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:07:08.603629    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:07:09.278810 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:07:09.278446    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:07:09.278810 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:07:09.278514    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:07:09.424609 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:07:09.424556    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:07:09.424974 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:07:09.424932    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:07:09.719398 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:07:09.719331    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:07:21.326836 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:07:21.326799    4614 scope.go:122] "RemoveContainer" containerID="0f529e052082ca789e79880f46ec132e83119defa18d1a4c70fe5386842e990c"
Nov 02 10:07:21.335439 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:07:21.335417    4614 scope.go:122] "RemoveContainer" containerID="af8a00109bb63a01109bd46e9203c8ec6d3426125f4075e415fc8d371b2726d0"
Nov 02 10:07:39.634699 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:07:39.634661    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:08:06.200376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:08:06.199459    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:08:06.200376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:08:06.199530    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:08:09.638932 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:08:09.638895    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:08:39.635378 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:08:39.635314    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:09:06.207522 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:06.207428    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:09:06.208322 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:06.208291    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:09:09.296386 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:09.295454    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:09:09.296386 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:09.295539    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:09:09.426378 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:09.425484    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:09:09.426378 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:09.425586    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:09:09.678430 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:09:09.668438    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:09:21.367032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:21.366999    4614 scope.go:122] "RemoveContainer" containerID="6b0097d0a2d8bc97b7abdb33305fd2d4f723ffdd51f2d6e9e141ba3ff7c5e395"
Nov 02 10:09:23.367176 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.367140    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"]
Nov 02 10:09:23.367687 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.367551    4614 cpu_manager.go:414] "containerMap: removing container" podUID="7cc668fb-131c-424c-94e1-9c56c8a726dd" containerName="revoke-keys"
Nov 02 10:09:23.367687 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.367563    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc668fb-131c-424c-94e1-9c56c8a726dd" containerName="revoke-keys"
Nov 02 10:09:23.367687 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.367621    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc668fb-131c-424c-94e1-9c56c8a726dd" containerName="revoke-keys"
Nov 02 10:09:23.372379 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.372344    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:23.382379 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.382334    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"]
Nov 02 10:09:23.383998 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.383974    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 10:09:23.383998 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.383974    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-dd40e2-serving-cert\""
Nov 02 10:09:23.384274 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.384249    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5thw\""
Nov 02 10:09:23.409660 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.409629    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/270b6ae0-1de2-465e-bf8b-7cb298b6be08-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:23.409851 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.409680    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh8fn\" (UniqueName: \"kubernetes.io/projected/270b6ae0-1de2-465e-bf8b-7cb298b6be08-kube-api-access-rh8fn\") pod \"maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:23.409851 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.409754    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/270b6ae0-1de2-465e-bf8b-7cb298b6be08-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:23.409851 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.409797    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270b6ae0-1de2-465e-bf8b-7cb298b6be08-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:23.511301 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.511264    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270b6ae0-1de2-465e-bf8b-7cb298b6be08-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:23.511491 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.511397    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/270b6ae0-1de2-465e-bf8b-7cb298b6be08-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:23.511491 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.511459    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh8fn\" (UniqueName: \"kubernetes.io/projected/270b6ae0-1de2-465e-bf8b-7cb298b6be08-kube-api-access-rh8fn\") pod \"maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:23.511491 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.511485    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/270b6ae0-1de2-465e-bf8b-7cb298b6be08-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:23.512141 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.512117    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/270b6ae0-1de2-465e-bf8b-7cb298b6be08-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:23.514017 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.513999    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270b6ae0-1de2-465e-bf8b-7cb298b6be08-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:23.514115 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.514046    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/270b6ae0-1de2-465e-bf8b-7cb298b6be08-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:23.545026 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.544991    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh8fn\" (UniqueName: \"kubernetes.io/projected/270b6ae0-1de2-465e-bf8b-7cb298b6be08-kube-api-access-rh8fn\") pod \"maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:23.689645 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.689561    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:23.972427 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:23.972049    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"]
Nov 02 10:09:23.987127 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:09:23.987092    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270b6ae0_1de2_465e_bf8b_7cb298b6be08.slice/crio-64d7b2b39de3956d395527f904f4fa374438065450fda0469bd0daf9d5ef9eb2 WatchSource:0}: Error finding container 64d7b2b39de3956d395527f904f4fa374438065450fda0469bd0daf9d5ef9eb2: Status 404 returned error can't find the container with id 64d7b2b39de3956d395527f904f4fa374438065450fda0469bd0daf9d5ef9eb2
Nov 02 10:09:24.623841 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:24.623813    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm" event={"ID":"270b6ae0-1de2-465e-bf8b-7cb298b6be08","Type":"ContainerStarted","Data":"17c3aed12f21746016d94db1de2c6c59356d6e5a04e14b95b16ff75daf484bbe"}
Nov 02 10:09:24.624187 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:24.623864    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm" event={"ID":"270b6ae0-1de2-465e-bf8b-7cb298b6be08","Type":"ContainerStarted","Data":"64d7b2b39de3956d395527f904f4fa374438065450fda0469bd0daf9d5ef9eb2"}
Nov 02 10:09:24.624187 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:24.623945    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:24.644219 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:24.644172    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm" podStartSLOduration=1.072141898 podStartE2EDuration="1.644159579s" totalImagesPullingTime="572.017681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:09:24.642528869 +0000 UTC m=+2045.475186949" watchObservedRunningTime="2026-11-02 10:09:24.644159579 +0000 UTC m=+2045.476817673"
Nov 02 10:09:30.634776 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:09:30.634748    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:09:39.636529 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:09:39.636493    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:10:09.635765 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:10:09.635726    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:10:19.830033 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:19.829985    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:10:19.855764 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:19.855741    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:10:21.188911 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:21.188873    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"]
Nov 02 10:10:21.189405 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:21.189116    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm" podUID="270b6ae0-1de2-465e-bf8b-7cb298b6be08" containerName="maas-api" containerID="cri-o://17c3aed12f21746016d94db1de2c6c59356d6e5a04e14b95b16ff75daf484bbe" gracePeriod=30
Nov 02 10:10:22.448270 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.448244    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:10:22.472269 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.472223    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270b6ae0-1de2-465e-bf8b-7cb298b6be08-kube-api-access-rh8fn\" (UniqueName: \"kubernetes.io/projected/270b6ae0-1de2-465e-bf8b-7cb298b6be08-kube-api-access-rh8fn\") pod \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") "
Nov 02 10:10:22.472269 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.472265    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/270b6ae0-1de2-465e-bf8b-7cb298b6be08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/270b6ae0-1de2-465e-bf8b-7cb298b6be08-openshift-service-ca\") pod \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") "
Nov 02 10:10:22.472516 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.472328    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/270b6ae0-1de2-465e-bf8b-7cb298b6be08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/270b6ae0-1de2-465e-bf8b-7cb298b6be08-metrics-certs\") pod \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") "
Nov 02 10:10:22.472516 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.472430    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/270b6ae0-1de2-465e-bf8b-7cb298b6be08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/270b6ae0-1de2-465e-bf8b-7cb298b6be08-maas-api-tls\") pod \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\" (UID: \"270b6ae0-1de2-465e-bf8b-7cb298b6be08\") "
Nov 02 10:10:22.472741 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.472715    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270b6ae0-1de2-465e-bf8b-7cb298b6be08-openshift-service-ca" pod "270b6ae0-1de2-465e-bf8b-7cb298b6be08" (UID: "270b6ae0-1de2-465e-bf8b-7cb298b6be08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:10:22.475079 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.475050    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270b6ae0-1de2-465e-bf8b-7cb298b6be08-maas-api-tls" pod "270b6ae0-1de2-465e-bf8b-7cb298b6be08" (UID: "270b6ae0-1de2-465e-bf8b-7cb298b6be08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:10:22.475241 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.475145    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270b6ae0-1de2-465e-bf8b-7cb298b6be08-metrics-certs" pod "270b6ae0-1de2-465e-bf8b-7cb298b6be08" (UID: "270b6ae0-1de2-465e-bf8b-7cb298b6be08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:10:22.475861 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.475825    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270b6ae0-1de2-465e-bf8b-7cb298b6be08-kube-api-access-rh8fn" pod "270b6ae0-1de2-465e-bf8b-7cb298b6be08" (UID: "270b6ae0-1de2-465e-bf8b-7cb298b6be08"). InnerVolumeSpecName "kube-api-access-rh8fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:10:22.573758 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.573671    4614 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270b6ae0-1de2-465e-bf8b-7cb298b6be08-metrics-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:22.573758 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.573701    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/270b6ae0-1de2-465e-bf8b-7cb298b6be08-maas-api-tls\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:22.573758 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.573715    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh8fn\" (UniqueName: \"kubernetes.io/projected/270b6ae0-1de2-465e-bf8b-7cb298b6be08-kube-api-access-rh8fn\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:22.573758 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.573733    4614 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/270b6ae0-1de2-465e-bf8b-7cb298b6be08-openshift-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:10:22.912709 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.912614    4614 generic.go:356] "Generic (PLEG): container finished" podID="270b6ae0-1de2-465e-bf8b-7cb298b6be08" containerID="17c3aed12f21746016d94db1de2c6c59356d6e5a04e14b95b16ff75daf484bbe" exitCode=0
Nov 02 10:10:22.912709 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.912683    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"
Nov 02 10:10:22.912949 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.912697    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm" event={"ID":"270b6ae0-1de2-465e-bf8b-7cb298b6be08","Type":"ContainerDied","Data":"17c3aed12f21746016d94db1de2c6c59356d6e5a04e14b95b16ff75daf484bbe"}
Nov 02 10:10:22.912949 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.912781    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm" event={"ID":"270b6ae0-1de2-465e-bf8b-7cb298b6be08","Type":"ContainerDied","Data":"64d7b2b39de3956d395527f904f4fa374438065450fda0469bd0daf9d5ef9eb2"}
Nov 02 10:10:22.912949 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.912805    4614 scope.go:122] "RemoveContainer" containerID="17c3aed12f21746016d94db1de2c6c59356d6e5a04e14b95b16ff75daf484bbe"
Nov 02 10:10:22.922293 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.922274    4614 scope.go:122] "RemoveContainer" containerID="17c3aed12f21746016d94db1de2c6c59356d6e5a04e14b95b16ff75daf484bbe"
Nov 02 10:10:22.922573 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:10:22.922554    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c3aed12f21746016d94db1de2c6c59356d6e5a04e14b95b16ff75daf484bbe\": container with ID starting with 17c3aed12f21746016d94db1de2c6c59356d6e5a04e14b95b16ff75daf484bbe not found: ID does not exist" containerID="17c3aed12f21746016d94db1de2c6c59356d6e5a04e14b95b16ff75daf484bbe"
Nov 02 10:10:22.922656 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.922579    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c3aed12f21746016d94db1de2c6c59356d6e5a04e14b95b16ff75daf484bbe"} err="failed to get container status \"17c3aed12f21746016d94db1de2c6c59356d6e5a04e14b95b16ff75daf484bbe\": rpc error: code = NotFound desc = could not find container \"17c3aed12f21746016d94db1de2c6c59356d6e5a04e14b95b16ff75daf484bbe\": container with ID starting with 17c3aed12f21746016d94db1de2c6c59356d6e5a04e14b95b16ff75daf484bbe not found: ID does not exist"
Nov 02 10:10:22.934948 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.934926    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"]
Nov 02 10:10:22.939247 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:22.939222    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dd40e2-6d5b74779f-rdwgm"]
Nov 02 10:10:23.321943 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:23.321902    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="270b6ae0-1de2-465e-bf8b-7cb298b6be08" path="/var/lib/kubelet/pods/270b6ae0-1de2-465e-bf8b-7cb298b6be08/volumes"
Nov 02 10:10:39.635017 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:10:39.634976    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:10:53.594377 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:53.593465    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:10:53.594377 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:53.593538    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:10:54.282378 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:54.276689    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:10:54.282378 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:54.276762    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:10:54.424104 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:54.423574    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:10:54.424104 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:10:54.423659    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:11:09.786375 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:11:09.785530    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:11:17.023989 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:11:17.023953    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"]
Nov 02 10:11:17.026622 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:11:17.026596    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-t8nv6"]
Nov 02 10:11:17.314464 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:11:17.314390    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55107c99-4d99-4ac4-83eb-b9563f43aea1" path="/var/lib/kubelet/pods/55107c99-4d99-4ac4-83eb-b9563f43aea1/volumes"
Nov 02 10:11:38.598380 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:11:38.596475    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:11:38.598380 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:11:38.596557    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:11:39.280377 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:11:39.279875    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:11:39.280377 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:11:39.279954    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:11:39.659249 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:11:39.659105    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:11:53.598511 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:11:53.598470    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:11:53.599203 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:11:53.599169    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:11:56.024490 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:11:56.024450    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"]
Nov 02 10:11:56.032288 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:11:56.032255    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-c9kwh"]
Nov 02 10:11:57.334272 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:11:57.334234    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc668fb-131c-424c-94e1-9c56c8a726dd" path="/var/lib/kubelet/pods/7cc668fb-131c-424c-94e1-9c56c8a726dd/volumes"
Nov 02 10:12:09.635840 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:12:09.635753    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:12:21.407470 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:12:21.407438    4614 scope.go:122] "RemoveContainer" containerID="dfb8be980a9eebacde607d232eb1677174506c7bf88173ed4efbd30e543bb3ef"
Nov 02 10:12:21.415923 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:12:21.415899    4614 scope.go:122] "RemoveContainer" containerID="3e076bf794f6b924ba6e56418efcb702d97adf10e8c83eac5ddbc7b3bdbbdac2"
Nov 02 10:12:39.635972 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:12:39.635928    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:13:09.635335 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:13:09.635297    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:13:15.205145 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.205107    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"]
Nov 02 10:13:15.205795 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.205771    4614 cpu_manager.go:414] "containerMap: removing container" podUID="270b6ae0-1de2-465e-bf8b-7cb298b6be08" containerName="maas-api"
Nov 02 10:13:15.205891 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.205797    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="270b6ae0-1de2-465e-bf8b-7cb298b6be08" containerName="maas-api"
Nov 02 10:13:15.205955 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.205938    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="270b6ae0-1de2-465e-bf8b-7cb298b6be08" containerName="maas-api"
Nov 02 10:13:15.209184 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.209158    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.215705 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.215683    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2b5a18\"/\"openshift-service-ca.crt\""
Nov 02 10:13:15.216011 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.215989    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2b5a18\"/\"kube-root-ca.crt\""
Nov 02 10:13:15.216845 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.216824    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2b5a18\"/\"ipp-route-2b5a18-kserve-self-signed-certs\""
Nov 02 10:13:15.216946 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.216932    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2b5a18\"/\"default-dockercfg-c9h9v\""
Nov 02 10:13:15.224630 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.224608    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"]
Nov 02 10:13:15.351679 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.351645    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p772\" (UniqueName: \"kubernetes.io/projected/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-kube-api-access-6p772\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.351855 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.351697    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-home\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.351855 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.351774    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-model-cache\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.351855 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.351805    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-tls-certs\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.351855 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.351831    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-dshm\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.352001 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.351909    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-tmp-dir\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.453010 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.452969    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p772\" (UniqueName: \"kubernetes.io/projected/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-kube-api-access-6p772\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.453219 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.453077    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-home\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.453219 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.453104    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-model-cache\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.453219 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.453129    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-tls-certs\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.453219 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.453158    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-dshm\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.453479 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.453254    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-tmp-dir\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.453549 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.453529    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-home\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.453614 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.453582    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-model-cache\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.454177 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.454155    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-tmp-dir\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.455587 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.455535    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-dshm\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.456007 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.455990    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-tls-certs\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.465873 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.465840    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p772\" (UniqueName: \"kubernetes.io/projected/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-kube-api-access-6p772\") pod \"ipp-route-2b5a18-kserve-7ccd7b675c-9l95x\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") " pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.528368 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.528328    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:15.809275 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:15.809249    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"]
Nov 02 10:13:15.811622 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:13:15.811593    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ff935c5_1be1_4fc6_83a5_18bdf0aeb721.slice/crio-bbdd7f900495951e50c923dd74db0def4ccc9c5b3beec3214ead5c80ed7321fc WatchSource:0}: Error finding container bbdd7f900495951e50c923dd74db0def4ccc9c5b3beec3214ead5c80ed7321fc: Status 404 returned error can't find the container with id bbdd7f900495951e50c923dd74db0def4ccc9c5b3beec3214ead5c80ed7321fc
Nov 02 10:13:16.022575 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:16.022535    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x" event={"ID":"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721","Type":"ContainerStarted","Data":"0900be8d41a2294acc7d81cadd8ccc8479fcc5a37e87fba0f9e24e471e79ff2d"}
Nov 02 10:13:16.022746 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:16.022589    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x" event={"ID":"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721","Type":"ContainerStarted","Data":"bbdd7f900495951e50c923dd74db0def4ccc9c5b3beec3214ead5c80ed7321fc"}
Nov 02 10:13:16.042005 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:16.041959    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x" podStartSLOduration=1.041946334 podStartE2EDuration="1.041946334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:13:16.04093164 +0000 UTC m=+2276.873589720" watchObservedRunningTime="2026-11-02 10:13:16.041946334 +0000 UTC m=+2276.874604414"
Nov 02 10:13:16.529132 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:16.529102    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:21.438958 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:21.438925    4614 scope.go:122] "RemoveContainer" containerID="ccd0ad18d049ec6d892e19e158d52f45555af7c54a669e8adea8683aaf160640"
Nov 02 10:13:25.529326 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:25.529285    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:25.542203 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:25.542180    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:26.103864 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:26.103836    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:13:39.634923 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:13:39.634882    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:13:55.089051 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:13:55.089016    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"]
Nov 02 10:14:09.635535 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:14:09.635492    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:14:10.108892 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.108831    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x" podUID="0ff935c5-1be1-4fc6-83a5-18bdf0aeb721" containerName="main" containerID="cri-o://0900be8d41a2294acc7d81cadd8ccc8479fcc5a37e87fba0f9e24e471e79ff2d" gracePeriod=45
Nov 02 10:14:10.306831 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.306784    4614 generic.go:356] "Generic (PLEG): container finished" podID="0ff935c5-1be1-4fc6-83a5-18bdf0aeb721" containerID="0900be8d41a2294acc7d81cadd8ccc8479fcc5a37e87fba0f9e24e471e79ff2d" exitCode=0
Nov 02 10:14:10.307022 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.306932    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x" event={"ID":"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721","Type":"ContainerDied","Data":"0900be8d41a2294acc7d81cadd8ccc8479fcc5a37e87fba0f9e24e471e79ff2d"}
Nov 02 10:14:10.365063 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.364996    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:14:10.465183 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.465152    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-model-cache\") pod \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") "
Nov 02 10:14:10.465396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.465241    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-tmp-dir\") pod \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") "
Nov 02 10:14:10.465396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.465288    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-dshm\") pod \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") "
Nov 02 10:14:10.465396 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.465379    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-home\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-home\") pod \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") "
Nov 02 10:14:10.465580 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.465406    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-tls-certs\") pod \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") "
Nov 02 10:14:10.465580 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.465431    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-kube-api-access-6p772\" (UniqueName: \"kubernetes.io/projected/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-kube-api-access-6p772\") pod \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\" (UID: \"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721\") "
Nov 02 10:14:10.465580 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.465502    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-model-cache" pod "0ff935c5-1be1-4fc6-83a5-18bdf0aeb721" (UID: "0ff935c5-1be1-4fc6-83a5-18bdf0aeb721"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:14:10.465736 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.465581    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-tmp-dir" pod "0ff935c5-1be1-4fc6-83a5-18bdf0aeb721" (UID: "0ff935c5-1be1-4fc6-83a5-18bdf0aeb721"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:14:10.465736 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.465709    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-home" pod "0ff935c5-1be1-4fc6-83a5-18bdf0aeb721" (UID: "0ff935c5-1be1-4fc6-83a5-18bdf0aeb721"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:14:10.465843 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.465817    4614 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-home\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:10.465951 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.465931    4614 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-model-cache\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:10.466009 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.465958    4614 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-tmp-dir\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:10.467666 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.467641    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-dshm" pod "0ff935c5-1be1-4fc6-83a5-18bdf0aeb721" (UID: "0ff935c5-1be1-4fc6-83a5-18bdf0aeb721"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:14:10.467666 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.467651    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-tls-certs" pod "0ff935c5-1be1-4fc6-83a5-18bdf0aeb721" (UID: "0ff935c5-1be1-4fc6-83a5-18bdf0aeb721"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:14:10.467814 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.467691    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-kube-api-access-6p772" pod "0ff935c5-1be1-4fc6-83a5-18bdf0aeb721" (UID: "0ff935c5-1be1-4fc6-83a5-18bdf0aeb721"). InnerVolumeSpecName "kube-api-access-6p772". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:14:10.566913 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.566881    4614 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-tls-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:10.566913 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.566908    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p772\" (UniqueName: \"kubernetes.io/projected/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-kube-api-access-6p772\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:10.566913 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:10.566919    4614 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721-dshm\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:11.312332 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:11.312307    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"
Nov 02 10:14:11.314111 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:11.314082    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x" event={"ID":"0ff935c5-1be1-4fc6-83a5-18bdf0aeb721","Type":"ContainerDied","Data":"bbdd7f900495951e50c923dd74db0def4ccc9c5b3beec3214ead5c80ed7321fc"}
Nov 02 10:14:11.314185 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:11.314131    4614 scope.go:122] "RemoveContainer" containerID="0900be8d41a2294acc7d81cadd8ccc8479fcc5a37e87fba0f9e24e471e79ff2d"
Nov 02 10:14:11.337192 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:11.337163    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"]
Nov 02 10:14:11.342928 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:11.342905    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2b5a18/ipp-route-2b5a18-kserve-7ccd7b675c-9l95x"]
Nov 02 10:14:13.314266 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:13.314226    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff935c5-1be1-4fc6-83a5-18bdf0aeb721" path="/var/lib/kubelet/pods/0ff935c5-1be1-4fc6-83a5-18bdf0aeb721/volumes"
Nov 02 10:14:32.499488 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.499453    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"]
Nov 02 10:14:32.499973 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.499827    4614 cpu_manager.go:414] "containerMap: removing container" podUID="0ff935c5-1be1-4fc6-83a5-18bdf0aeb721" containerName="main"
Nov 02 10:14:32.499973 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.499839    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff935c5-1be1-4fc6-83a5-18bdf0aeb721" containerName="main"
Nov 02 10:14:32.499973 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.499888    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff935c5-1be1-4fc6-83a5-18bdf0aeb721" containerName="main"
Nov 02 10:14:32.504123 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.504107    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"
Nov 02 10:14:32.511096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.511054    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 10:14:32.511232 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.511125    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:14:32.513683 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.513657    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"]
Nov 02 10:14:32.567974 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.567944    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4a31b1-h6lbt\" (UID: \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"
Nov 02 10:14:32.568128 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.567979    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4a31b1-h6lbt\" (UID: \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"
Nov 02 10:14:32.568128 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.568082    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5prj9\" (UniqueName: \"kubernetes.io/projected/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-kube-api-access-5prj9\") pod \"maas-api-revoke-sub-3a4a31b1-h6lbt\" (UID: \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"
Nov 02 10:14:32.669090 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.669061    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5prj9\" (UniqueName: \"kubernetes.io/projected/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-kube-api-access-5prj9\") pod \"maas-api-revoke-sub-3a4a31b1-h6lbt\" (UID: \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"
Nov 02 10:14:32.669242 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.669118    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4a31b1-h6lbt\" (UID: \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"
Nov 02 10:14:32.669285 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.669231    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4a31b1-h6lbt\" (UID: \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"
Nov 02 10:14:32.669740 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.669720    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4a31b1-h6lbt\" (UID: \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"
Nov 02 10:14:32.671528 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.671508    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4a31b1-h6lbt\" (UID: \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"
Nov 02 10:14:32.681603 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.681580    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5prj9\" (UniqueName: \"kubernetes.io/projected/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-kube-api-access-5prj9\") pod \"maas-api-revoke-sub-3a4a31b1-h6lbt\" (UID: \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"
Nov 02 10:14:32.819526 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:32.819458    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"
Nov 02 10:14:33.111889 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:33.111860    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"]
Nov 02 10:14:33.113799 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:14:33.113772    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9ec5ea_e770_44ba_9a2a_76ba7f1787de.slice/crio-7b45049ffabbae747554c3eb829136059eb489c1361de0feb254d6f64ca6f214 WatchSource:0}: Error finding container 7b45049ffabbae747554c3eb829136059eb489c1361de0feb254d6f64ca6f214: Status 404 returned error can't find the container with id 7b45049ffabbae747554c3eb829136059eb489c1361de0feb254d6f64ca6f214
Nov 02 10:14:33.414981 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:33.414904    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt" event={"ID":"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de","Type":"ContainerStarted","Data":"7b45049ffabbae747554c3eb829136059eb489c1361de0feb254d6f64ca6f214"}
Nov 02 10:14:34.420745 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:34.420706    4614 generic.go:356] "Generic (PLEG): container finished" podID="bc9ec5ea-e770-44ba-9a2a-76ba7f1787de" containerID="0fbc40290a34ecb02f51e7d5614ca32f069495620f2a81b3cf10cc5b362c35e6" exitCode=0
Nov 02 10:14:34.421146 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:34.420795    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt" event={"ID":"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de","Type":"ContainerDied","Data":"0fbc40290a34ecb02f51e7d5614ca32f069495620f2a81b3cf10cc5b362c35e6"}
Nov 02 10:14:35.557649 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:35.557625    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"
Nov 02 10:14:35.599848 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:35.599809    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-maas-api-openshift-ca\") pod \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\" (UID: \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\") "
Nov 02 10:14:35.600095 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:35.599859    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-maas-api-xks-ca\") pod \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\" (UID: \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\") "
Nov 02 10:14:35.600095 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:35.599902    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-kube-api-access-5prj9\" (UniqueName: \"kubernetes.io/projected/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-kube-api-access-5prj9\") pod \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\" (UID: \"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de\") "
Nov 02 10:14:35.600245 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:35.600146    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-maas-api-openshift-ca" pod "bc9ec5ea-e770-44ba-9a2a-76ba7f1787de" (UID: "bc9ec5ea-e770-44ba-9a2a-76ba7f1787de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:14:35.601980 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:35.601952    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-maas-api-xks-ca" pod "bc9ec5ea-e770-44ba-9a2a-76ba7f1787de" (UID: "bc9ec5ea-e770-44ba-9a2a-76ba7f1787de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:14:35.602114 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:35.602000    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-kube-api-access-5prj9" pod "bc9ec5ea-e770-44ba-9a2a-76ba7f1787de" (UID: "bc9ec5ea-e770-44ba-9a2a-76ba7f1787de"). InnerVolumeSpecName "kube-api-access-5prj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:14:35.701241 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:35.701155    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-maas-api-openshift-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:35.701241 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:35.701190    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-maas-api-xks-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:35.701241 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:35.701199    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5prj9\" (UniqueName: \"kubernetes.io/projected/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de-kube-api-access-5prj9\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:14:36.431203 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:36.431172    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"
Nov 02 10:14:36.431410 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:36.431172    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt" event={"ID":"bc9ec5ea-e770-44ba-9a2a-76ba7f1787de","Type":"ContainerDied","Data":"7b45049ffabbae747554c3eb829136059eb489c1361de0feb254d6f64ca6f214"}
Nov 02 10:14:36.431410 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:14:36.431298    4614 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b45049ffabbae747554c3eb829136059eb489c1361de0feb254d6f64ca6f214"
Nov 02 10:14:39.634317 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:14:39.634278    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:15:00.138271 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.138189    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"]
Nov 02 10:15:00.138646 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.138607    4614 cpu_manager.go:414] "containerMap: removing container" podUID="bc9ec5ea-e770-44ba-9a2a-76ba7f1787de" containerName="revoke-keys"
Nov 02 10:15:00.138646 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.138620    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9ec5ea-e770-44ba-9a2a-76ba7f1787de" containerName="revoke-keys"
Nov 02 10:15:00.138719 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.138683    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9ec5ea-e770-44ba-9a2a-76ba7f1787de" containerName="revoke-keys"
Nov 02 10:15:00.141891 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.141874    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"
Nov 02 10:15:00.150046 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.150016    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"]
Nov 02 10:15:00.151940 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.151919    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fq4p2\""
Nov 02 10:15:00.245437 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.245405    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qcwh\" (UniqueName: \"kubernetes.io/projected/f11d4157-7da5-4fb0-873d-7ae145e4261b-kube-api-access-6qcwh\") pod \"maas-api-key-cleanup-29893575-w8hnz\" (UID: \"f11d4157-7da5-4fb0-873d-7ae145e4261b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"
Nov 02 10:15:00.245437 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.245440    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11d4157-7da5-4fb0-873d-7ae145e4261b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893575-w8hnz\" (UID: \"f11d4157-7da5-4fb0-873d-7ae145e4261b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"
Nov 02 10:15:00.347044 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.347007    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qcwh\" (UniqueName: \"kubernetes.io/projected/f11d4157-7da5-4fb0-873d-7ae145e4261b-kube-api-access-6qcwh\") pod \"maas-api-key-cleanup-29893575-w8hnz\" (UID: \"f11d4157-7da5-4fb0-873d-7ae145e4261b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"
Nov 02 10:15:00.347044 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.347042    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11d4157-7da5-4fb0-873d-7ae145e4261b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893575-w8hnz\" (UID: \"f11d4157-7da5-4fb0-873d-7ae145e4261b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"
Nov 02 10:15:00.347664 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.347645    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11d4157-7da5-4fb0-873d-7ae145e4261b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893575-w8hnz\" (UID: \"f11d4157-7da5-4fb0-873d-7ae145e4261b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"
Nov 02 10:15:00.362110 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.362081    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qcwh\" (UniqueName: \"kubernetes.io/projected/f11d4157-7da5-4fb0-873d-7ae145e4261b-kube-api-access-6qcwh\") pod \"maas-api-key-cleanup-29893575-w8hnz\" (UID: \"f11d4157-7da5-4fb0-873d-7ae145e4261b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"
Nov 02 10:15:00.456555 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.456478    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"
Nov 02 10:15:00.732515 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:00.732383    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"]
Nov 02 10:15:00.735313 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:15:00.735282    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11d4157_7da5_4fb0_873d_7ae145e4261b.slice/crio-792de6cd7d987abfc72a2eb7db5d4a8294592d6cf64985310b5d103cb179cb9a WatchSource:0}: Error finding container 792de6cd7d987abfc72a2eb7db5d4a8294592d6cf64985310b5d103cb179cb9a: Status 404 returned error can't find the container with id 792de6cd7d987abfc72a2eb7db5d4a8294592d6cf64985310b5d103cb179cb9a
Nov 02 10:15:01.557505 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:01.557468    4614 generic.go:356] "Generic (PLEG): container finished" podID="f11d4157-7da5-4fb0-873d-7ae145e4261b" containerID="9c3fc78da41db9018e12652320b893ec7aead045c228a5ae17aee2da96f4635e" exitCode=60
Nov 02 10:15:01.557939 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:01.557540    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz" event={"ID":"f11d4157-7da5-4fb0-873d-7ae145e4261b","Type":"ContainerDied","Data":"9c3fc78da41db9018e12652320b893ec7aead045c228a5ae17aee2da96f4635e"}
Nov 02 10:15:01.557939 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:01.557592    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz" event={"ID":"f11d4157-7da5-4fb0-873d-7ae145e4261b","Type":"ContainerStarted","Data":"792de6cd7d987abfc72a2eb7db5d4a8294592d6cf64985310b5d103cb179cb9a"}
Nov 02 10:15:01.557939 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:01.557868    4614 scope.go:122] "RemoveContainer" containerID="9c3fc78da41db9018e12652320b893ec7aead045c228a5ae17aee2da96f4635e"
Nov 02 10:15:02.564334 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:02.564304    4614 generic.go:356] "Generic (PLEG): container finished" podID="f11d4157-7da5-4fb0-873d-7ae145e4261b" containerID="e4f4685d3be3c0e3096f9b3f65da267b08611fd34b19d0060bc48a960d8afaf5" exitCode=60
Nov 02 10:15:02.564768 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:02.564346    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz" event={"ID":"f11d4157-7da5-4fb0-873d-7ae145e4261b","Type":"ContainerDied","Data":"e4f4685d3be3c0e3096f9b3f65da267b08611fd34b19d0060bc48a960d8afaf5"}
Nov 02 10:15:02.564768 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:02.564430    4614 scope.go:122] "RemoveContainer" containerID="9c3fc78da41db9018e12652320b893ec7aead045c228a5ae17aee2da96f4635e"
Nov 02 10:15:02.564768 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:02.564669    4614 scope.go:122] "RemoveContainer" containerID="e4f4685d3be3c0e3096f9b3f65da267b08611fd34b19d0060bc48a960d8afaf5"
Nov 02 10:15:02.564927 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:15:02.564899    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893575-w8hnz_odh-ai-gateway-infra(f11d4157-7da5-4fb0-873d-7ae145e4261b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz" podUID="f11d4157-7da5-4fb0-873d-7ae145e4261b"
Nov 02 10:15:03.571387 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:03.571338    4614 scope.go:122] "RemoveContainer" containerID="e4f4685d3be3c0e3096f9b3f65da267b08611fd34b19d0060bc48a960d8afaf5"
Nov 02 10:15:03.571848 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:15:03.571575    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893575-w8hnz_odh-ai-gateway-infra(f11d4157-7da5-4fb0-873d-7ae145e4261b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz" podUID="f11d4157-7da5-4fb0-873d-7ae145e4261b"
Nov 02 10:15:09.634525 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:15:09.634485    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:15:15.308841 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:15.308809    4614 scope.go:122] "RemoveContainer" containerID="e4f4685d3be3c0e3096f9b3f65da267b08611fd34b19d0060bc48a960d8afaf5"
Nov 02 10:15:15.640174 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:15.640090    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz" event={"ID":"f11d4157-7da5-4fb0-873d-7ae145e4261b","Type":"ContainerStarted","Data":"fecb5ceb0dd17a38bc933a473c53893ea37fbf5bf4d6050e579a79568d8e79d1"}
Nov 02 10:15:15.659585 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:15.659532    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz" podStartSLOduration=15.659516231 podStartE2EDuration="15.659516231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:15:15.658180382 +0000 UTC m=+2396.490838461" watchObservedRunningTime="2026-11-02 10:15:15.659516231 +0000 UTC m=+2396.492174310"
Nov 02 10:15:16.647456 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:16.647421    4614 generic.go:356] "Generic (PLEG): container finished" podID="f11d4157-7da5-4fb0-873d-7ae145e4261b" containerID="fecb5ceb0dd17a38bc933a473c53893ea37fbf5bf4d6050e579a79568d8e79d1" exitCode=60
Nov 02 10:15:16.647898 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:16.647494    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz" event={"ID":"f11d4157-7da5-4fb0-873d-7ae145e4261b","Type":"ContainerDied","Data":"fecb5ceb0dd17a38bc933a473c53893ea37fbf5bf4d6050e579a79568d8e79d1"}
Nov 02 10:15:16.647898 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:16.647577    4614 scope.go:122] "RemoveContainer" containerID="e4f4685d3be3c0e3096f9b3f65da267b08611fd34b19d0060bc48a960d8afaf5"
Nov 02 10:15:16.647898 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:16.647876    4614 scope.go:122] "RemoveContainer" containerID="fecb5ceb0dd17a38bc933a473c53893ea37fbf5bf4d6050e579a79568d8e79d1"
Nov 02 10:15:16.648110 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:15:16.648088    4614 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893575-w8hnz_odh-ai-gateway-infra(f11d4157-7da5-4fb0-873d-7ae145e4261b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz" podUID="f11d4157-7da5-4fb0-873d-7ae145e4261b"
Nov 02 10:15:16.675762 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:16.675722    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"]
Nov 02 10:15:17.789249 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:17.789221    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"
Nov 02 10:15:17.919006 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:17.918918    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f11d4157-7da5-4fb0-873d-7ae145e4261b-kube-api-access-6qcwh\" (UniqueName: \"kubernetes.io/projected/f11d4157-7da5-4fb0-873d-7ae145e4261b-kube-api-access-6qcwh\") pod \"f11d4157-7da5-4fb0-873d-7ae145e4261b\" (UID: \"f11d4157-7da5-4fb0-873d-7ae145e4261b\") "
Nov 02 10:15:17.919165 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:17.919113    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f11d4157-7da5-4fb0-873d-7ae145e4261b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11d4157-7da5-4fb0-873d-7ae145e4261b-maas-api-service-ca\") pod \"f11d4157-7da5-4fb0-873d-7ae145e4261b\" (UID: \"f11d4157-7da5-4fb0-873d-7ae145e4261b\") "
Nov 02 10:15:17.919486 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:17.919462    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f11d4157-7da5-4fb0-873d-7ae145e4261b-maas-api-service-ca" pod "f11d4157-7da5-4fb0-873d-7ae145e4261b" (UID: "f11d4157-7da5-4fb0-873d-7ae145e4261b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:15:17.921082 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:17.921063    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f11d4157-7da5-4fb0-873d-7ae145e4261b-kube-api-access-6qcwh" pod "f11d4157-7da5-4fb0-873d-7ae145e4261b" (UID: "f11d4157-7da5-4fb0-873d-7ae145e4261b"). InnerVolumeSpecName "kube-api-access-6qcwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:15:18.020333 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:18.020297    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qcwh\" (UniqueName: \"kubernetes.io/projected/f11d4157-7da5-4fb0-873d-7ae145e4261b-kube-api-access-6qcwh\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:18.020333 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:18.020331    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11d4157-7da5-4fb0-873d-7ae145e4261b-maas-api-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:15:18.661551 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:18.661517    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"
Nov 02 10:15:18.661766 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:18.661559    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz" event={"ID":"f11d4157-7da5-4fb0-873d-7ae145e4261b","Type":"ContainerDied","Data":"792de6cd7d987abfc72a2eb7db5d4a8294592d6cf64985310b5d103cb179cb9a"}
Nov 02 10:15:18.661766 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:18.661620    4614 scope.go:122] "RemoveContainer" containerID="fecb5ceb0dd17a38bc933a473c53893ea37fbf5bf4d6050e579a79568d8e79d1"
Nov 02 10:15:18.686561 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:18.686528    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"]
Nov 02 10:15:18.692243 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:18.692219    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893575-w8hnz"]
Nov 02 10:15:19.314198 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:19.314163    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f11d4157-7da5-4fb0-873d-7ae145e4261b" path="/var/lib/kubelet/pods/f11d4157-7da5-4fb0-873d-7ae145e4261b/volumes"
Nov 02 10:15:19.857313 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:19.857285    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:15:19.883989 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:15:19.883964    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:15:39.635162 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:15:39.635123    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:16:09.634318 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:16:09.634278    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:16:39.635062 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:16:39.635025    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:17:09.635124 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:17:09.635086    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:17:39.280498 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:17:39.280443    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:17:39.281246 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:17:39.281201    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:17:39.424123 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:17:39.423473    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:17:39.424123 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:17:39.423583    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:17:39.847375 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:17:39.845661    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:17:53.598377 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:17:53.597745    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:17:53.598377 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:17:53.597829    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:17:54.301043 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:17:54.301004    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:17:54.301508 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:17:54.301476    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:17:54.425590 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:17:54.425547    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:17:54.425964 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:17:54.425934    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:18:09.712430 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:18:09.712169    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:18:39.634297 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:18:39.634262    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:19:09.635164 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:19:09.635127    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:19:36.024440 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:19:36.024346    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"]
Nov 02 10:19:36.026773 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:19:36.026740    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4a31b1-h6lbt"]
Nov 02 10:19:37.318286 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:19:37.318251    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9ec5ea-e770-44ba-9a2a-76ba7f1787de" path="/var/lib/kubelet/pods/bc9ec5ea-e770-44ba-9a2a-76ba7f1787de/volumes"
Nov 02 10:19:38.600529 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:19:38.600480    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5msqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:19:38.601183 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:19:38.601146    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:19:39.634074 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:19:39.634034    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:20:09.635813 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:20:09.635768    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:20:19.966376 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:20:19.958407    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:20:20.021668 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:20:20.021636    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:20:21.192524 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:20:21.192478    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:20:21.193202 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:20:21.193161    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:20:39.285378 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:20:39.284451    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:20:39.285378 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:20:39.284517    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:20:39.420649 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:20:39.420605    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:20:39.421022 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:20:39.420982    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:20:39.726485 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:20:39.726392    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:21:09.635154 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:21:09.635066    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:21:21.189582 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:21:21.189521    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:21:21.192523 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:21:21.190191    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:21:21.530377 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:21:21.526912    4614 scope.go:122] "RemoveContainer" containerID="0fbc40290a34ecb02f51e7d5614ca32f069495620f2a81b3cf10cc5b362c35e6"
Nov 02 10:21:39.635083 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:21:39.635045    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:22:09.634796 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:22:09.634758    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:22:39.635181 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:22:39.635143    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:23:09.636346 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:23:09.636304    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:23:39.820378 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:23:39.816765    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:24:09.635051 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:24:09.634972    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:24:39.638556 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:24:39.638515    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:25:09.635917 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:25:09.635880    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:25:20.164152 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:20.164099    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:25:20.213494 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:20.213463    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:25:21.225387 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:21.221492    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:25:21.225387 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:21.221579    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:25:34.959938 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:34.959907    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"]
Nov 02 10:25:34.960385 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:34.960269    4614 cpu_manager.go:414] "containerMap: removing container" podUID="f11d4157-7da5-4fb0-873d-7ae145e4261b" containerName="cleanup"
Nov 02 10:25:34.960385 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:34.960281    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f11d4157-7da5-4fb0-873d-7ae145e4261b" containerName="cleanup"
Nov 02 10:25:34.960385 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:34.960293    4614 cpu_manager.go:414] "containerMap: removing container" podUID="f11d4157-7da5-4fb0-873d-7ae145e4261b" containerName="cleanup"
Nov 02 10:25:34.960385 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:34.960298    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f11d4157-7da5-4fb0-873d-7ae145e4261b" containerName="cleanup"
Nov 02 10:25:34.960385 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:34.960310    4614 cpu_manager.go:414] "containerMap: removing container" podUID="f11d4157-7da5-4fb0-873d-7ae145e4261b" containerName="cleanup"
Nov 02 10:25:34.960385 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:34.960315    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f11d4157-7da5-4fb0-873d-7ae145e4261b" containerName="cleanup"
Nov 02 10:25:34.960603 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:34.960412    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="f11d4157-7da5-4fb0-873d-7ae145e4261b" containerName="cleanup"
Nov 02 10:25:34.960603 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:34.960424    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="f11d4157-7da5-4fb0-873d-7ae145e4261b" containerName="cleanup"
Nov 02 10:25:34.963597 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:34.963575    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:34.971199 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:34.971165    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5thw\""
Nov 02 10:25:34.973390 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:34.973333    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 10:25:34.973583 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:34.973561    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 10:25:34.975104 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:34.975074    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"]
Nov 02 10:25:35.070604 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.070559    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lssnr\" (UniqueName: \"kubernetes.io/projected/9756cee1-7732-4acf-a7b9-70f547cf1b6e-kube-api-access-lssnr\") pod \"maas-api-7ff9894dcf-nqs4f\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:35.070604 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.070605    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9756cee1-7732-4acf-a7b9-70f547cf1b6e-maas-api-tls\") pod \"maas-api-7ff9894dcf-nqs4f\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:35.070844 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.070684    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9756cee1-7732-4acf-a7b9-70f547cf1b6e-openshift-service-ca\") pod \"maas-api-7ff9894dcf-nqs4f\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:35.070844 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.070759    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9756cee1-7732-4acf-a7b9-70f547cf1b6e-metrics-certs\") pod \"maas-api-7ff9894dcf-nqs4f\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:35.171897 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.171866    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9756cee1-7732-4acf-a7b9-70f547cf1b6e-metrics-certs\") pod \"maas-api-7ff9894dcf-nqs4f\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:35.172109 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.171922    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lssnr\" (UniqueName: \"kubernetes.io/projected/9756cee1-7732-4acf-a7b9-70f547cf1b6e-kube-api-access-lssnr\") pod \"maas-api-7ff9894dcf-nqs4f\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:35.172109 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.171957    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9756cee1-7732-4acf-a7b9-70f547cf1b6e-maas-api-tls\") pod \"maas-api-7ff9894dcf-nqs4f\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:35.172109 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.171992    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9756cee1-7732-4acf-a7b9-70f547cf1b6e-openshift-service-ca\") pod \"maas-api-7ff9894dcf-nqs4f\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:35.172733 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.172713    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9756cee1-7732-4acf-a7b9-70f547cf1b6e-openshift-service-ca\") pod \"maas-api-7ff9894dcf-nqs4f\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:35.174393 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.174374    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9756cee1-7732-4acf-a7b9-70f547cf1b6e-maas-api-tls\") pod \"maas-api-7ff9894dcf-nqs4f\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:35.174579 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.174560    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9756cee1-7732-4acf-a7b9-70f547cf1b6e-metrics-certs\") pod \"maas-api-7ff9894dcf-nqs4f\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:35.183914 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.183893    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lssnr\" (UniqueName: \"kubernetes.io/projected/9756cee1-7732-4acf-a7b9-70f547cf1b6e-kube-api-access-lssnr\") pod \"maas-api-7ff9894dcf-nqs4f\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") " pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:35.280024 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.279989    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:35.555246 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.555169    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"]
Nov 02 10:25:35.558093 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:25:35.558064    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9756cee1_7732_4acf_a7b9_70f547cf1b6e.slice/crio-0d6886c86ac74a55e713a8dbb798eca326cb0830cd2ba626f4e9b3edd3fb3bdd WatchSource:0}: Error finding container 0d6886c86ac74a55e713a8dbb798eca326cb0830cd2ba626f4e9b3edd3fb3bdd: Status 404 returned error can't find the container with id 0d6886c86ac74a55e713a8dbb798eca326cb0830cd2ba626f4e9b3edd3fb3bdd
Nov 02 10:25:35.559475 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:35.559452    4614 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 10:25:36.227787 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:36.227752    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f" event={"ID":"9756cee1-7732-4acf-a7b9-70f547cf1b6e","Type":"ContainerStarted","Data":"9b7ebfa42a85cfbaedbbefb22910b8948117b0422a6c4665b93b7166055c16a2"}
Nov 02 10:25:36.228294 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:36.227811    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f" event={"ID":"9756cee1-7732-4acf-a7b9-70f547cf1b6e","Type":"ContainerStarted","Data":"0d6886c86ac74a55e713a8dbb798eca326cb0830cd2ba626f4e9b3edd3fb3bdd"}
Nov 02 10:25:36.228294 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:36.227893    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:36.250007 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:36.249960    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f" podStartSLOduration=1.7122341159999999 podStartE2EDuration="2.24994882s" totalImagesPullingTime="537.714704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:25:36.248169233 +0000 UTC m=+3017.080827328" watchObservedRunningTime="2026-11-02 10:25:36.24994882 +0000 UTC m=+3017.082606899"
Nov 02 10:25:39.635874 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:25:39.635840    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:25:42.238441 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.238412    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:42.382790 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.382757    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"]
Nov 02 10:25:42.383182 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.383170    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="f11d4157-7da5-4fb0-873d-7ae145e4261b" containerName="cleanup"
Nov 02 10:25:42.386298 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.386283    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:42.397871 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.397849    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"]
Nov 02 10:25:42.433891 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.433852    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d5bf9bb-32fa-4d3e-9f35-b050565e4782-metrics-certs\") pod \"maas-api-6647fcb89f-kzxjl\" (UID: \"0d5bf9bb-32fa-4d3e-9f35-b050565e4782\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:42.434046 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.433908    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5bf9bb-32fa-4d3e-9f35-b050565e4782-openshift-service-ca\") pod \"maas-api-6647fcb89f-kzxjl\" (UID: \"0d5bf9bb-32fa-4d3e-9f35-b050565e4782\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:42.434046 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.434014    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mnzz\" (UniqueName: \"kubernetes.io/projected/0d5bf9bb-32fa-4d3e-9f35-b050565e4782-kube-api-access-6mnzz\") pod \"maas-api-6647fcb89f-kzxjl\" (UID: \"0d5bf9bb-32fa-4d3e-9f35-b050565e4782\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:42.434135 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.434062    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d5bf9bb-32fa-4d3e-9f35-b050565e4782-maas-api-tls\") pod \"maas-api-6647fcb89f-kzxjl\" (UID: \"0d5bf9bb-32fa-4d3e-9f35-b050565e4782\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:42.534997 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.534909    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d5bf9bb-32fa-4d3e-9f35-b050565e4782-metrics-certs\") pod \"maas-api-6647fcb89f-kzxjl\" (UID: \"0d5bf9bb-32fa-4d3e-9f35-b050565e4782\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:42.534997 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.534954    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5bf9bb-32fa-4d3e-9f35-b050565e4782-openshift-service-ca\") pod \"maas-api-6647fcb89f-kzxjl\" (UID: \"0d5bf9bb-32fa-4d3e-9f35-b050565e4782\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:42.535265 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.535003    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mnzz\" (UniqueName: \"kubernetes.io/projected/0d5bf9bb-32fa-4d3e-9f35-b050565e4782-kube-api-access-6mnzz\") pod \"maas-api-6647fcb89f-kzxjl\" (UID: \"0d5bf9bb-32fa-4d3e-9f35-b050565e4782\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:42.535265 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.535055    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d5bf9bb-32fa-4d3e-9f35-b050565e4782-maas-api-tls\") pod \"maas-api-6647fcb89f-kzxjl\" (UID: \"0d5bf9bb-32fa-4d3e-9f35-b050565e4782\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:42.535662 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.535640    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d5bf9bb-32fa-4d3e-9f35-b050565e4782-openshift-service-ca\") pod \"maas-api-6647fcb89f-kzxjl\" (UID: \"0d5bf9bb-32fa-4d3e-9f35-b050565e4782\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:42.537461 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.537436    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d5bf9bb-32fa-4d3e-9f35-b050565e4782-metrics-certs\") pod \"maas-api-6647fcb89f-kzxjl\" (UID: \"0d5bf9bb-32fa-4d3e-9f35-b050565e4782\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:42.537612 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.537591    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d5bf9bb-32fa-4d3e-9f35-b050565e4782-maas-api-tls\") pod \"maas-api-6647fcb89f-kzxjl\" (UID: \"0d5bf9bb-32fa-4d3e-9f35-b050565e4782\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:42.547090 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.547069    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mnzz\" (UniqueName: \"kubernetes.io/projected/0d5bf9bb-32fa-4d3e-9f35-b050565e4782-kube-api-access-6mnzz\") pod \"maas-api-6647fcb89f-kzxjl\" (UID: \"0d5bf9bb-32fa-4d3e-9f35-b050565e4782\") " pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:42.700445 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.700413    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:42.976904 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:42.976866    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"]
Nov 02 10:25:42.979192 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:25:42.979167    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5bf9bb_32fa_4d3e_9f35_b050565e4782.slice/crio-d16e91ff7c33a33e485edf7d95a5e881d46835911c8401b4d9959008f4d17c53 WatchSource:0}: Error finding container d16e91ff7c33a33e485edf7d95a5e881d46835911c8401b4d9959008f4d17c53: Status 404 returned error can't find the container with id d16e91ff7c33a33e485edf7d95a5e881d46835911c8401b4d9959008f4d17c53
Nov 02 10:25:43.265061 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:43.264980    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl" event={"ID":"0d5bf9bb-32fa-4d3e-9f35-b050565e4782","Type":"ContainerStarted","Data":"d16e91ff7c33a33e485edf7d95a5e881d46835911c8401b4d9959008f4d17c53"}
Nov 02 10:25:44.274096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:44.274047    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl" event={"ID":"0d5bf9bb-32fa-4d3e-9f35-b050565e4782","Type":"ContainerStarted","Data":"720e9117efb5508b15805418c3647908dadc4e153ecc50cfc585fc71780cfdb6"}
Nov 02 10:25:44.274682 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:44.274145    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:44.296654 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:44.296580    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl" podStartSLOduration=1.90408379 podStartE2EDuration="2.296558941s" totalImagesPullingTime="392.475151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:25:44.293320725 +0000 UTC m=+3025.125978806" watchObservedRunningTime="2026-11-02 10:25:44.296558941 +0000 UTC m=+3025.129217022"
Nov 02 10:25:50.291803 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.291773    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6647fcb89f-kzxjl"
Nov 02 10:25:50.345515 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.345482    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"]
Nov 02 10:25:50.345797 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.345772    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f" podUID="9756cee1-7732-4acf-a7b9-70f547cf1b6e" containerName="maas-api" containerID="cri-o://9b7ebfa42a85cfbaedbbefb22910b8948117b0422a6c4665b93b7166055c16a2" gracePeriod=30
Nov 02 10:25:50.581906 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.581880    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:50.676083 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.676055    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9756cee1-7732-4acf-a7b9-70f547cf1b6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9756cee1-7732-4acf-a7b9-70f547cf1b6e-maas-api-tls\") pod \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") "
Nov 02 10:25:50.676280 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.676131    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9756cee1-7732-4acf-a7b9-70f547cf1b6e-kube-api-access-lssnr\" (UniqueName: \"kubernetes.io/projected/9756cee1-7732-4acf-a7b9-70f547cf1b6e-kube-api-access-lssnr\") pod \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") "
Nov 02 10:25:50.676280 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.676158    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9756cee1-7732-4acf-a7b9-70f547cf1b6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9756cee1-7732-4acf-a7b9-70f547cf1b6e-metrics-certs\") pod \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") "
Nov 02 10:25:50.676280 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.676183    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9756cee1-7732-4acf-a7b9-70f547cf1b6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9756cee1-7732-4acf-a7b9-70f547cf1b6e-openshift-service-ca\") pod \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\" (UID: \"9756cee1-7732-4acf-a7b9-70f547cf1b6e\") "
Nov 02 10:25:50.676674 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.676643    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9756cee1-7732-4acf-a7b9-70f547cf1b6e-openshift-service-ca" pod "9756cee1-7732-4acf-a7b9-70f547cf1b6e" (UID: "9756cee1-7732-4acf-a7b9-70f547cf1b6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:25:50.678364 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.678328    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9756cee1-7732-4acf-a7b9-70f547cf1b6e-metrics-certs" pod "9756cee1-7732-4acf-a7b9-70f547cf1b6e" (UID: "9756cee1-7732-4acf-a7b9-70f547cf1b6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:25:50.678461 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.678400    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9756cee1-7732-4acf-a7b9-70f547cf1b6e-kube-api-access-lssnr" pod "9756cee1-7732-4acf-a7b9-70f547cf1b6e" (UID: "9756cee1-7732-4acf-a7b9-70f547cf1b6e"). InnerVolumeSpecName "kube-api-access-lssnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:25:50.678508 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.678467    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9756cee1-7732-4acf-a7b9-70f547cf1b6e-maas-api-tls" pod "9756cee1-7732-4acf-a7b9-70f547cf1b6e" (UID: "9756cee1-7732-4acf-a7b9-70f547cf1b6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 10:25:50.777421 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.777380    4614 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9756cee1-7732-4acf-a7b9-70f547cf1b6e-openshift-service-ca\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:25:50.777421 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.777415    4614 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9756cee1-7732-4acf-a7b9-70f547cf1b6e-maas-api-tls\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:25:50.777421 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.777426    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lssnr\" (UniqueName: \"kubernetes.io/projected/9756cee1-7732-4acf-a7b9-70f547cf1b6e-kube-api-access-lssnr\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:25:50.777671 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:50.777436    4614 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9756cee1-7732-4acf-a7b9-70f547cf1b6e-metrics-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:25:51.362671 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:51.362635    4614 generic.go:356] "Generic (PLEG): container finished" podID="9756cee1-7732-4acf-a7b9-70f547cf1b6e" containerID="9b7ebfa42a85cfbaedbbefb22910b8948117b0422a6c4665b93b7166055c16a2" exitCode=0
Nov 02 10:25:51.363045 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:51.362692    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"
Nov 02 10:25:51.363045 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:51.362718    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f" event={"ID":"9756cee1-7732-4acf-a7b9-70f547cf1b6e","Type":"ContainerDied","Data":"9b7ebfa42a85cfbaedbbefb22910b8948117b0422a6c4665b93b7166055c16a2"}
Nov 02 10:25:51.363045 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:51.362799    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f" event={"ID":"9756cee1-7732-4acf-a7b9-70f547cf1b6e","Type":"ContainerDied","Data":"0d6886c86ac74a55e713a8dbb798eca326cb0830cd2ba626f4e9b3edd3fb3bdd"}
Nov 02 10:25:51.363045 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:51.362826    4614 scope.go:122] "RemoveContainer" containerID="9b7ebfa42a85cfbaedbbefb22910b8948117b0422a6c4665b93b7166055c16a2"
Nov 02 10:25:51.371535 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:51.371517    4614 scope.go:122] "RemoveContainer" containerID="9b7ebfa42a85cfbaedbbefb22910b8948117b0422a6c4665b93b7166055c16a2"
Nov 02 10:25:51.371799 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:25:51.371780    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7ebfa42a85cfbaedbbefb22910b8948117b0422a6c4665b93b7166055c16a2\": container with ID starting with 9b7ebfa42a85cfbaedbbefb22910b8948117b0422a6c4665b93b7166055c16a2 not found: ID does not exist" containerID="9b7ebfa42a85cfbaedbbefb22910b8948117b0422a6c4665b93b7166055c16a2"
Nov 02 10:25:51.371853 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:51.371807    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7ebfa42a85cfbaedbbefb22910b8948117b0422a6c4665b93b7166055c16a2"} err="failed to get container status \"9b7ebfa42a85cfbaedbbefb22910b8948117b0422a6c4665b93b7166055c16a2\": rpc error: code = NotFound desc = could not find container \"9b7ebfa42a85cfbaedbbefb22910b8948117b0422a6c4665b93b7166055c16a2\": container with ID starting with 9b7ebfa42a85cfbaedbbefb22910b8948117b0422a6c4665b93b7166055c16a2 not found: ID does not exist"
Nov 02 10:25:51.380680 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:51.380654    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"]
Nov 02 10:25:51.385566 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:51.385543    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff9894dcf-nqs4f"]
Nov 02 10:25:53.314156 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:25:53.314121    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9756cee1-7732-4acf-a7b9-70f547cf1b6e" path="/var/lib/kubelet/pods/9756cee1-7732-4acf-a7b9-70f547cf1b6e/volumes"
Nov 02 10:26:09.634841 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:26:09.634795    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:26:39.635679 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:26:39.635635    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:27:09.635150 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:27:09.635057    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:27:39.634939 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:27:39.634898    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:28:09.635051 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:28:09.635006    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:28:39.634864 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:28:39.634825    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:29:09.634799 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:29:09.634762    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:29:39.634839 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:29:39.634801    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:30:09.635123 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:30:09.635033    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:30:20.260317 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:30:20.260290    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:30:20.301739 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:30:20.301705    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:30:39.635241 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:30:39.635203    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:31:09.635858 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:31:09.635816    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:31:11.230742 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:11.230704    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"]
Nov 02 10:31:11.231253 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:11.231002    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy" containerID="cri-o://39d45fa80450268c3ffbc1333293fb1c27afc5b5922a27455028b6ed1d2ae256" gracePeriod=30
Nov 02 10:31:11.233875 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:11.233847    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"]
Nov 02 10:31:11.234079 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:11.234049    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy" containerID="cri-o://5540f746aa4e4da7f6f99005d4113e79a76ebe9baed54fd957bad95c37b42e50" gracePeriod=30
Nov 02 10:31:11.235602 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:11.235561    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"]
Nov 02 10:31:11.235924 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:11.235894    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy" containerID="cri-o://990910fd6fe246a46f0dd8124ad3fb0e82cc2e5d6397e1625ec878381dd9b4a3" gracePeriod=30
Nov 02 10:31:11.237240 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:11.237214    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"]
Nov 02 10:31:11.237467 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:11.237440    4614 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy" containerID="cri-o://24d9804661f73184e1c2ebe886e02c653a50120f2b1f6a64e01658cc1f9aa1c9" gracePeriod=30
Nov 02 10:31:16.514769 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.514375    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:31:16.566989 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.566963    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:31:16.570203 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.570182    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:31:16.573676 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.573659    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 10:31:16.590445 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.590409    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-workload-certs\") pod \"67fe0167-799a-4caa-9318-ef604f0ca331\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") "
Nov 02 10:31:16.590603 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.590451    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-istio-data\") pod \"67fe0167-799a-4caa-9318-ef604f0ca331\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") "
Nov 02 10:31:16.590603 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.590512    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-istio-envoy\") pod \"67fe0167-799a-4caa-9318-ef604f0ca331\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") "
Nov 02 10:31:16.590603 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.590585    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-workload-socket\") pod \"67fe0167-799a-4caa-9318-ef604f0ca331\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") "
Nov 02 10:31:16.590805 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.590617    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67fe0167-799a-4caa-9318-ef604f0ca331-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67fe0167-799a-4caa-9318-ef604f0ca331-istiod-ca-cert\") pod \"67fe0167-799a-4caa-9318-ef604f0ca331\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") "
Nov 02 10:31:16.590805 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.590672    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-credential-socket\") pod \"67fe0167-799a-4caa-9318-ef604f0ca331\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") "
Nov 02 10:31:16.590805 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.590725    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67fe0167-799a-4caa-9318-ef604f0ca331-istio-token\" (UniqueName: \"kubernetes.io/projected/67fe0167-799a-4caa-9318-ef604f0ca331-istio-token\") pod \"67fe0167-799a-4caa-9318-ef604f0ca331\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") "
Nov 02 10:31:16.590805 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.590756    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67fe0167-799a-4caa-9318-ef604f0ca331-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67fe0167-799a-4caa-9318-ef604f0ca331-istio-podinfo\") pod \"67fe0167-799a-4caa-9318-ef604f0ca331\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") "
Nov 02 10:31:16.590805 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.590794    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67fe0167-799a-4caa-9318-ef604f0ca331-kube-api-access-dcscw\" (UniqueName: \"kubernetes.io/projected/67fe0167-799a-4caa-9318-ef604f0ca331-kube-api-access-dcscw\") pod \"67fe0167-799a-4caa-9318-ef604f0ca331\" (UID: \"67fe0167-799a-4caa-9318-ef604f0ca331\") "
Nov 02 10:31:16.591224 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.591172    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-workload-socket" pod "67fe0167-799a-4caa-9318-ef604f0ca331" (UID: "67fe0167-799a-4caa-9318-ef604f0ca331"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.591503 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.591480    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-credential-socket" pod "67fe0167-799a-4caa-9318-ef604f0ca331" (UID: "67fe0167-799a-4caa-9318-ef604f0ca331"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.591587 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.591496    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67fe0167-799a-4caa-9318-ef604f0ca331-istiod-ca-cert" pod "67fe0167-799a-4caa-9318-ef604f0ca331" (UID: "67fe0167-799a-4caa-9318-ef604f0ca331"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:31:16.591700 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.591681    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-workload-certs" pod "67fe0167-799a-4caa-9318-ef604f0ca331" (UID: "67fe0167-799a-4caa-9318-ef604f0ca331"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.592231 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.592198    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-istio-data" pod "67fe0167-799a-4caa-9318-ef604f0ca331" (UID: "67fe0167-799a-4caa-9318-ef604f0ca331"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.593717 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.593672    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fe0167-799a-4caa-9318-ef604f0ca331-istio-token" pod "67fe0167-799a-4caa-9318-ef604f0ca331" (UID: "67fe0167-799a-4caa-9318-ef604f0ca331"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:31:16.594246 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.594220    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fe0167-799a-4caa-9318-ef604f0ca331-kube-api-access-dcscw" pod "67fe0167-799a-4caa-9318-ef604f0ca331" (UID: "67fe0167-799a-4caa-9318-ef604f0ca331"). InnerVolumeSpecName "kube-api-access-dcscw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:31:16.594474 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.594442    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67fe0167-799a-4caa-9318-ef604f0ca331-istio-podinfo" pod "67fe0167-799a-4caa-9318-ef604f0ca331" (UID: "67fe0167-799a-4caa-9318-ef604f0ca331"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:31:16.595735 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.595712    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-istio-envoy" pod "67fe0167-799a-4caa-9318-ef604f0ca331" (UID: "67fe0167-799a-4caa-9318-ef604f0ca331"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.692065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692022    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-envoy\") pod \"69ff7ab2-6bde-4cad-8459-8fad2688572b\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") "
Nov 02 10:31:16.692286 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692086    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-workload-certs\") pod \"14493b0f-4cb8-474a-ae5d-216823f6d2be\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") "
Nov 02 10:31:16.692286 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692128    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-podinfo\") pod \"d57e7275-fe40-4a86-b6e9-0939c91526b0\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") "
Nov 02 10:31:16.692286 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692159    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14493b0f-4cb8-474a-ae5d-216823f6d2be-kube-api-access-jn6gf\" (UniqueName: \"kubernetes.io/projected/14493b0f-4cb8-474a-ae5d-216823f6d2be-kube-api-access-jn6gf\") pod \"14493b0f-4cb8-474a-ae5d-216823f6d2be\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") "
Nov 02 10:31:16.692286 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692203    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-workload-socket\") pod \"14493b0f-4cb8-474a-ae5d-216823f6d2be\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") "
Nov 02 10:31:16.692286 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692233    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-credential-socket\") pod \"69ff7ab2-6bde-4cad-8459-8fad2688572b\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") "
Nov 02 10:31:16.692695 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692343    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-credential-socket\") pod \"d57e7275-fe40-4a86-b6e9-0939c91526b0\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") "
Nov 02 10:31:16.692695 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692413    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d57e7275-fe40-4a86-b6e9-0939c91526b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57e7275-fe40-4a86-b6e9-0939c91526b0-istiod-ca-cert\") pod \"d57e7275-fe40-4a86-b6e9-0939c91526b0\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") "
Nov 02 10:31:16.692695 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692433    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-workload-certs" pod "14493b0f-4cb8-474a-ae5d-216823f6d2be" (UID: "14493b0f-4cb8-474a-ae5d-216823f6d2be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.692695 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692464    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-token\" (UniqueName: \"kubernetes.io/projected/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-token\") pod \"d57e7275-fe40-4a86-b6e9-0939c91526b0\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") "
Nov 02 10:31:16.692695 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692509    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-data\") pod \"69ff7ab2-6bde-4cad-8459-8fad2688572b\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") "
Nov 02 10:31:16.692695 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692521    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-credential-socket" pod "69ff7ab2-6bde-4cad-8459-8fad2688572b" (UID: "69ff7ab2-6bde-4cad-8459-8fad2688572b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.692695 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692557    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ff7ab2-6bde-4cad-8459-8fad2688572b-kube-api-access-dbkj7\" (UniqueName: \"kubernetes.io/projected/69ff7ab2-6bde-4cad-8459-8fad2688572b-kube-api-access-dbkj7\") pod \"69ff7ab2-6bde-4cad-8459-8fad2688572b\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") "
Nov 02 10:31:16.692695 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692593    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ff7ab2-6bde-4cad-8459-8fad2688572b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ff7ab2-6bde-4cad-8459-8fad2688572b-istiod-ca-cert\") pod \"69ff7ab2-6bde-4cad-8459-8fad2688572b\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") "
Nov 02 10:31:16.692695 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692629    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-data\") pod \"14493b0f-4cb8-474a-ae5d-216823f6d2be\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") "
Nov 02 10:31:16.692695 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692659    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-credential-socket\") pod \"14493b0f-4cb8-474a-ae5d-216823f6d2be\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") "
Nov 02 10:31:16.692695 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692687    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-data\") pod \"d57e7275-fe40-4a86-b6e9-0939c91526b0\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") "
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692713    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-credential-socket" pod "d57e7275-fe40-4a86-b6e9-0939c91526b0" (UID: "d57e7275-fe40-4a86-b6e9-0939c91526b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692736    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-token\" (UniqueName: \"kubernetes.io/projected/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-token\") pod \"14493b0f-4cb8-474a-ae5d-216823f6d2be\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") "
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692805    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-podinfo\") pod \"69ff7ab2-6bde-4cad-8459-8fad2688572b\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") "
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692850    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-podinfo\") pod \"14493b0f-4cb8-474a-ae5d-216823f6d2be\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") "
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692883    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57e7275-fe40-4a86-b6e9-0939c91526b0-istiod-ca-cert" pod "d57e7275-fe40-4a86-b6e9-0939c91526b0" (UID: "d57e7275-fe40-4a86-b6e9-0939c91526b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692901    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-workload-certs\") pod \"69ff7ab2-6bde-4cad-8459-8fad2688572b\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") "
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692948    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-workload-socket\") pod \"d57e7275-fe40-4a86-b6e9-0939c91526b0\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") "
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.692988    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-envoy\") pod \"d57e7275-fe40-4a86-b6e9-0939c91526b0\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") "
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.693022    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-workload-socket\") pod \"69ff7ab2-6bde-4cad-8459-8fad2688572b\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") "
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.693056    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-workload-certs\") pod \"d57e7275-fe40-4a86-b6e9-0939c91526b0\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") "
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.693084    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-envoy\") pod \"14493b0f-4cb8-474a-ae5d-216823f6d2be\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") "
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.693109    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-workload-socket" pod "14493b0f-4cb8-474a-ae5d-216823f6d2be" (UID: "14493b0f-4cb8-474a-ae5d-216823f6d2be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.693114    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57e7275-fe40-4a86-b6e9-0939c91526b0-kube-api-access-4k5vn\" (UniqueName: \"kubernetes.io/projected/d57e7275-fe40-4a86-b6e9-0939c91526b0-kube-api-access-4k5vn\") pod \"d57e7275-fe40-4a86-b6e9-0939c91526b0\" (UID: \"d57e7275-fe40-4a86-b6e9-0939c91526b0\") "
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.693171    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-token\" (UniqueName: \"kubernetes.io/projected/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-token\") pod \"69ff7ab2-6bde-4cad-8459-8fad2688572b\" (UID: \"69ff7ab2-6bde-4cad-8459-8fad2688572b\") "
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.693236    4614 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14493b0f-4cb8-474a-ae5d-216823f6d2be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14493b0f-4cb8-474a-ae5d-216823f6d2be-istiod-ca-cert\") pod \"14493b0f-4cb8-474a-ae5d-216823f6d2be\" (UID: \"14493b0f-4cb8-474a-ae5d-216823f6d2be\") "
Nov 02 10:31:16.693276 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.693242    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-credential-socket" pod "14493b0f-4cb8-474a-ae5d-216823f6d2be" (UID: "14493b0f-4cb8-474a-ae5d-216823f6d2be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.694638 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.693417    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-data" pod "14493b0f-4cb8-474a-ae5d-216823f6d2be" (UID: "14493b0f-4cb8-474a-ae5d-216823f6d2be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.694638 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.693445    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-data" pod "d57e7275-fe40-4a86-b6e9-0939c91526b0" (UID: "d57e7275-fe40-4a86-b6e9-0939c91526b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.694638 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.693726    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14493b0f-4cb8-474a-ae5d-216823f6d2be-istiod-ca-cert" pod "14493b0f-4cb8-474a-ae5d-216823f6d2be" (UID: "14493b0f-4cb8-474a-ae5d-216823f6d2be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:31:16.694638 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.693810    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-data" pod "69ff7ab2-6bde-4cad-8459-8fad2688572b" (UID: "69ff7ab2-6bde-4cad-8459-8fad2688572b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.694638 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.694189    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.694638 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.694214    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695068 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695045    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14493b0f-4cb8-474a-ae5d-216823f6d2be-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695149 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695074    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695149 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695101    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695149 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695118    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67fe0167-799a-4caa-9318-ef604f0ca331-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695149 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695133    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695149    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695165    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695180    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695195    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d57e7275-fe40-4a86-b6e9-0939c91526b0-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695210    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67fe0167-799a-4caa-9318-ef604f0ca331-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695224    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67fe0167-799a-4caa-9318-ef604f0ca331-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695239    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695254    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcscw\" (UniqueName: \"kubernetes.io/projected/67fe0167-799a-4caa-9318-ef604f0ca331-kube-api-access-dcscw\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695271    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695285    4614 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-credential-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695299    4614 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-data\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695314    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67fe0167-799a-4caa-9318-ef604f0ca331-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.694316    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-workload-socket" pod "69ff7ab2-6bde-4cad-8459-8fad2688572b" (UID: "69ff7ab2-6bde-4cad-8459-8fad2688572b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.694337    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-workload-certs" pod "69ff7ab2-6bde-4cad-8459-8fad2688572b" (UID: "69ff7ab2-6bde-4cad-8459-8fad2688572b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.694592    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-workload-certs" pod "d57e7275-fe40-4a86-b6e9-0939c91526b0" (UID: "d57e7275-fe40-4a86-b6e9-0939c91526b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.694776    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-workload-socket" pod "d57e7275-fe40-4a86-b6e9-0939c91526b0" (UID: "d57e7275-fe40-4a86-b6e9-0939c91526b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.695372 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695000    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ff7ab2-6bde-4cad-8459-8fad2688572b-istiod-ca-cert" pod "69ff7ab2-6bde-4cad-8459-8fad2688572b" (UID: "69ff7ab2-6bde-4cad-8459-8fad2688572b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 10:31:16.696434 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695536    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-podinfo" pod "69ff7ab2-6bde-4cad-8459-8fad2688572b" (UID: "69ff7ab2-6bde-4cad-8459-8fad2688572b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:31:16.696434 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.695699    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14493b0f-4cb8-474a-ae5d-216823f6d2be-kube-api-access-jn6gf" pod "14493b0f-4cb8-474a-ae5d-216823f6d2be" (UID: "14493b0f-4cb8-474a-ae5d-216823f6d2be"). InnerVolumeSpecName "kube-api-access-jn6gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:31:16.696434 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.696029    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57e7275-fe40-4a86-b6e9-0939c91526b0-kube-api-access-4k5vn" pod "d57e7275-fe40-4a86-b6e9-0939c91526b0" (UID: "d57e7275-fe40-4a86-b6e9-0939c91526b0"). InnerVolumeSpecName "kube-api-access-4k5vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:31:16.696434 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.696195    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-envoy" pod "69ff7ab2-6bde-4cad-8459-8fad2688572b" (UID: "69ff7ab2-6bde-4cad-8459-8fad2688572b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.696434 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.696417    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-token" pod "14493b0f-4cb8-474a-ae5d-216823f6d2be" (UID: "14493b0f-4cb8-474a-ae5d-216823f6d2be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:31:16.696720 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.696518    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-podinfo" pod "d57e7275-fe40-4a86-b6e9-0939c91526b0" (UID: "d57e7275-fe40-4a86-b6e9-0939c91526b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:31:16.697336 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.697300    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-envoy" pod "14493b0f-4cb8-474a-ae5d-216823f6d2be" (UID: "14493b0f-4cb8-474a-ae5d-216823f6d2be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.697477 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.697454    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-token" pod "69ff7ab2-6bde-4cad-8459-8fad2688572b" (UID: "69ff7ab2-6bde-4cad-8459-8fad2688572b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:31:16.697603 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.697586    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-envoy" pod "d57e7275-fe40-4a86-b6e9-0939c91526b0" (UID: "d57e7275-fe40-4a86-b6e9-0939c91526b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 10:31:16.697726 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.697694    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ff7ab2-6bde-4cad-8459-8fad2688572b-kube-api-access-dbkj7" pod "69ff7ab2-6bde-4cad-8459-8fad2688572b" (UID: "69ff7ab2-6bde-4cad-8459-8fad2688572b"). InnerVolumeSpecName "kube-api-access-dbkj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:31:16.697726 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.697714    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-token" pod "d57e7275-fe40-4a86-b6e9-0939c91526b0" (UID: "d57e7275-fe40-4a86-b6e9-0939c91526b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 10:31:16.698089 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.698071    4614 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-podinfo" pod "14493b0f-4cb8-474a-ae5d-216823f6d2be" (UID: "14493b0f-4cb8-474a-ae5d-216823f6d2be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 10:31:16.796564 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796490    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796564 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796516    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796564 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796525    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796564 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796534    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796564 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796543    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796564 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796552    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796564 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796561    4614 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-workload-socket\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796887 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796572    4614 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d57e7275-fe40-4a86-b6e9-0939c91526b0-workload-certs\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796887 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796580    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14493b0f-4cb8-474a-ae5d-216823f6d2be-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796887 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796590    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k5vn\" (UniqueName: \"kubernetes.io/projected/d57e7275-fe40-4a86-b6e9-0939c91526b0-kube-api-access-4k5vn\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796887 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796599    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796887 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796607    4614 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ff7ab2-6bde-4cad-8459-8fad2688572b-istio-envoy\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796887 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796616    4614 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-podinfo\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796887 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796624    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn6gf\" (UniqueName: \"kubernetes.io/projected/14493b0f-4cb8-474a-ae5d-216823f6d2be-kube-api-access-jn6gf\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796887 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796631    4614 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d57e7275-fe40-4a86-b6e9-0939c91526b0-istio-token\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796887 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796640    4614 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbkj7\" (UniqueName: \"kubernetes.io/projected/69ff7ab2-6bde-4cad-8459-8fad2688572b-kube-api-access-dbkj7\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.796887 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.796649    4614 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ff7ab2-6bde-4cad-8459-8fad2688572b-istiod-ca-cert\") on node \"ip-10-0-134-174.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 10:31:16.841092 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.841057    4614 generic.go:356] "Generic (PLEG): container finished" podID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerID="990910fd6fe246a46f0dd8124ad3fb0e82cc2e5d6397e1625ec878381dd9b4a3" exitCode=0
Nov 02 10:31:16.841281 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.841104    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"
Nov 02 10:31:16.841281 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.841142    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" event={"ID":"69ff7ab2-6bde-4cad-8459-8fad2688572b","Type":"ContainerDied","Data":"990910fd6fe246a46f0dd8124ad3fb0e82cc2e5d6397e1625ec878381dd9b4a3"}
Nov 02 10:31:16.841281 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.841204    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8" event={"ID":"69ff7ab2-6bde-4cad-8459-8fad2688572b","Type":"ContainerDied","Data":"91db7f4cdf22015bf3c402f72b0829f28745f7461e763f6a9295b7a0682864f8"}
Nov 02 10:31:16.841281 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.841228    4614 scope.go:122] "RemoveContainer" containerID="990910fd6fe246a46f0dd8124ad3fb0e82cc2e5d6397e1625ec878381dd9b4a3"
Nov 02 10:31:16.842583 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.842562    4614 generic.go:356] "Generic (PLEG): container finished" podID="67fe0167-799a-4caa-9318-ef604f0ca331" containerID="39d45fa80450268c3ffbc1333293fb1c27afc5b5922a27455028b6ed1d2ae256" exitCode=0
Nov 02 10:31:16.842697 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.842629    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"
Nov 02 10:31:16.842807 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.842615    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" event={"ID":"67fe0167-799a-4caa-9318-ef604f0ca331","Type":"ContainerDied","Data":"39d45fa80450268c3ffbc1333293fb1c27afc5b5922a27455028b6ed1d2ae256"}
Nov 02 10:31:16.842807 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.842762    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr" event={"ID":"67fe0167-799a-4caa-9318-ef604f0ca331","Type":"ContainerDied","Data":"803afbc5dc0789694d68e861ec5e6cafa2f7eaa8ad6f6c58102a5a56ed288712"}
Nov 02 10:31:16.844016 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.843995    4614 generic.go:356] "Generic (PLEG): container finished" podID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerID="5540f746aa4e4da7f6f99005d4113e79a76ebe9baed54fd957bad95c37b42e50" exitCode=0
Nov 02 10:31:16.844124 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.844109    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"
Nov 02 10:31:16.844222 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.844204    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" event={"ID":"d57e7275-fe40-4a86-b6e9-0939c91526b0","Type":"ContainerDied","Data":"5540f746aa4e4da7f6f99005d4113e79a76ebe9baed54fd957bad95c37b42e50"}
Nov 02 10:31:16.844284 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.844253    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt" event={"ID":"d57e7275-fe40-4a86-b6e9-0939c91526b0","Type":"ContainerDied","Data":"dab21dda3e40eceaec2f9ce9338ec72f108f763b0482618cfc61153ce79c1569"}
Nov 02 10:31:16.846110 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.846092    4614 generic.go:356] "Generic (PLEG): container finished" podID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerID="24d9804661f73184e1c2ebe886e02c653a50120f2b1f6a64e01658cc1f9aa1c9" exitCode=0
Nov 02 10:31:16.846191 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.846136    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" event={"ID":"14493b0f-4cb8-474a-ae5d-216823f6d2be","Type":"ContainerDied","Data":"24d9804661f73184e1c2ebe886e02c653a50120f2b1f6a64e01658cc1f9aa1c9"}
Nov 02 10:31:16.846191 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.846164    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx" event={"ID":"14493b0f-4cb8-474a-ae5d-216823f6d2be","Type":"ContainerDied","Data":"3a3a2c9d6ae1f29bc32917ecd1482be3555e04f9ebdf5b037587ef8169dc97c7"}
Nov 02 10:31:16.846262 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.846213    4614 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"
Nov 02 10:31:16.852482 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.852220    4614 scope.go:122] "RemoveContainer" containerID="990910fd6fe246a46f0dd8124ad3fb0e82cc2e5d6397e1625ec878381dd9b4a3"
Nov 02 10:31:16.852585 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:31:16.852563    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990910fd6fe246a46f0dd8124ad3fb0e82cc2e5d6397e1625ec878381dd9b4a3\": container with ID starting with 990910fd6fe246a46f0dd8124ad3fb0e82cc2e5d6397e1625ec878381dd9b4a3 not found: ID does not exist" containerID="990910fd6fe246a46f0dd8124ad3fb0e82cc2e5d6397e1625ec878381dd9b4a3"
Nov 02 10:31:16.852634 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.852597    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990910fd6fe246a46f0dd8124ad3fb0e82cc2e5d6397e1625ec878381dd9b4a3"} err="failed to get container status \"990910fd6fe246a46f0dd8124ad3fb0e82cc2e5d6397e1625ec878381dd9b4a3\": rpc error: code = NotFound desc = could not find container \"990910fd6fe246a46f0dd8124ad3fb0e82cc2e5d6397e1625ec878381dd9b4a3\": container with ID starting with 990910fd6fe246a46f0dd8124ad3fb0e82cc2e5d6397e1625ec878381dd9b4a3 not found: ID does not exist"
Nov 02 10:31:16.852634 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.852618    4614 scope.go:122] "RemoveContainer" containerID="39d45fa80450268c3ffbc1333293fb1c27afc5b5922a27455028b6ed1d2ae256"
Nov 02 10:31:16.862607 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.862586    4614 scope.go:122] "RemoveContainer" containerID="39d45fa80450268c3ffbc1333293fb1c27afc5b5922a27455028b6ed1d2ae256"
Nov 02 10:31:16.862888 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:31:16.862866    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d45fa80450268c3ffbc1333293fb1c27afc5b5922a27455028b6ed1d2ae256\": container with ID starting with 39d45fa80450268c3ffbc1333293fb1c27afc5b5922a27455028b6ed1d2ae256 not found: ID does not exist" containerID="39d45fa80450268c3ffbc1333293fb1c27afc5b5922a27455028b6ed1d2ae256"
Nov 02 10:31:16.862986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.862897    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d45fa80450268c3ffbc1333293fb1c27afc5b5922a27455028b6ed1d2ae256"} err="failed to get container status \"39d45fa80450268c3ffbc1333293fb1c27afc5b5922a27455028b6ed1d2ae256\": rpc error: code = NotFound desc = could not find container \"39d45fa80450268c3ffbc1333293fb1c27afc5b5922a27455028b6ed1d2ae256\": container with ID starting with 39d45fa80450268c3ffbc1333293fb1c27afc5b5922a27455028b6ed1d2ae256 not found: ID does not exist"
Nov 02 10:31:16.862986 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.862918    4614 scope.go:122] "RemoveContainer" containerID="5540f746aa4e4da7f6f99005d4113e79a76ebe9baed54fd957bad95c37b42e50"
Nov 02 10:31:16.871826 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.871805    4614 scope.go:122] "RemoveContainer" containerID="5540f746aa4e4da7f6f99005d4113e79a76ebe9baed54fd957bad95c37b42e50"
Nov 02 10:31:16.872092 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:31:16.872067    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5540f746aa4e4da7f6f99005d4113e79a76ebe9baed54fd957bad95c37b42e50\": container with ID starting with 5540f746aa4e4da7f6f99005d4113e79a76ebe9baed54fd957bad95c37b42e50 not found: ID does not exist" containerID="5540f746aa4e4da7f6f99005d4113e79a76ebe9baed54fd957bad95c37b42e50"
Nov 02 10:31:16.872219 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.872195    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5540f746aa4e4da7f6f99005d4113e79a76ebe9baed54fd957bad95c37b42e50"} err="failed to get container status \"5540f746aa4e4da7f6f99005d4113e79a76ebe9baed54fd957bad95c37b42e50\": rpc error: code = NotFound desc = could not find container \"5540f746aa4e4da7f6f99005d4113e79a76ebe9baed54fd957bad95c37b42e50\": container with ID starting with 5540f746aa4e4da7f6f99005d4113e79a76ebe9baed54fd957bad95c37b42e50 not found: ID does not exist"
Nov 02 10:31:16.872298 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.872221    4614 scope.go:122] "RemoveContainer" containerID="24d9804661f73184e1c2ebe886e02c653a50120f2b1f6a64e01658cc1f9aa1c9"
Nov 02 10:31:16.873929 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.873911    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"]
Nov 02 10:31:16.878245 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.878225    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sn2pt"]
Nov 02 10:31:16.880475 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.880457    4614 scope.go:122] "RemoveContainer" containerID="24d9804661f73184e1c2ebe886e02c653a50120f2b1f6a64e01658cc1f9aa1c9"
Nov 02 10:31:16.880750 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:31:16.880733    4614 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d9804661f73184e1c2ebe886e02c653a50120f2b1f6a64e01658cc1f9aa1c9\": container with ID starting with 24d9804661f73184e1c2ebe886e02c653a50120f2b1f6a64e01658cc1f9aa1c9 not found: ID does not exist" containerID="24d9804661f73184e1c2ebe886e02c653a50120f2b1f6a64e01658cc1f9aa1c9"
Nov 02 10:31:16.880796 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.880760    4614 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d9804661f73184e1c2ebe886e02c653a50120f2b1f6a64e01658cc1f9aa1c9"} err="failed to get container status \"24d9804661f73184e1c2ebe886e02c653a50120f2b1f6a64e01658cc1f9aa1c9\": rpc error: code = NotFound desc = could not find container \"24d9804661f73184e1c2ebe886e02c653a50120f2b1f6a64e01658cc1f9aa1c9\": container with ID starting with 24d9804661f73184e1c2ebe886e02c653a50120f2b1f6a64e01658cc1f9aa1c9 not found: ID does not exist"
Nov 02 10:31:16.889577 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.889557    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"]
Nov 02 10:31:16.895678 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.895659    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2lqv8"]
Nov 02 10:31:16.908114 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.908091    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"]
Nov 02 10:31:16.912731 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.912714    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5msqr"]
Nov 02 10:31:16.926028 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.926009    4614 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"]
Nov 02 10:31:16.932907 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:16.932886    4614 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-4rbgx"]
Nov 02 10:31:17.313676 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:17.313646    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" path="/var/lib/kubelet/pods/14493b0f-4cb8-474a-ae5d-216823f6d2be/volumes"
Nov 02 10:31:17.314077 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:17.314064    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" path="/var/lib/kubelet/pods/67fe0167-799a-4caa-9318-ef604f0ca331/volumes"
Nov 02 10:31:17.314461 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:17.314449    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" path="/var/lib/kubelet/pods/69ff7ab2-6bde-4cad-8459-8fad2688572b/volumes"
Nov 02 10:31:17.315156 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:31:17.315143    4614 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" path="/var/lib/kubelet/pods/d57e7275-fe40-4a86-b6e9-0939c91526b0/volumes"
Nov 02 10:31:39.635088 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:31:39.635040    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:32:09.635214 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:32:09.635175    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:32:39.635531 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:32:39.635494    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:33:09.634885 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:33:09.634806    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:33:11.380726 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.380693    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"]
Nov 02 10:33:11.381234 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381167    4614 cpu_manager.go:414] "containerMap: removing container" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy"
Nov 02 10:33:11.381234 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381186    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy"
Nov 02 10:33:11.381234 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381202    4614 cpu_manager.go:414] "containerMap: removing container" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy"
Nov 02 10:33:11.381234 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381210    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy"
Nov 02 10:33:11.381234 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381225    4614 cpu_manager.go:414] "containerMap: removing container" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy"
Nov 02 10:33:11.381234 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381232    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy"
Nov 02 10:33:11.381583 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381257    4614 cpu_manager.go:414] "containerMap: removing container" podUID="9756cee1-7732-4acf-a7b9-70f547cf1b6e" containerName="maas-api"
Nov 02 10:33:11.381583 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381265    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9756cee1-7732-4acf-a7b9-70f547cf1b6e" containerName="maas-api"
Nov 02 10:33:11.381583 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381289    4614 cpu_manager.go:414] "containerMap: removing container" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy"
Nov 02 10:33:11.381583 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381297    4614 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy"
Nov 02 10:33:11.381583 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381403    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="9756cee1-7732-4acf-a7b9-70f547cf1b6e" containerName="maas-api"
Nov 02 10:33:11.381583 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381421    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="67fe0167-799a-4caa-9318-ef604f0ca331" containerName="istio-proxy"
Nov 02 10:33:11.381583 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381434    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ff7ab2-6bde-4cad-8459-8fad2688572b" containerName="istio-proxy"
Nov 02 10:33:11.381583 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381443    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="14493b0f-4cb8-474a-ae5d-216823f6d2be" containerName="istio-proxy"
Nov 02 10:33:11.381583 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.381453    4614 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57e7275-fe40-4a86-b6e9-0939c91526b0" containerName="istio-proxy"
Nov 02 10:33:11.384685 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.384664    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.388789 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.388771    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b2fjp\""
Nov 02 10:33:11.400252 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.400231    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"]
Nov 02 10:33:11.494728 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.494693    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.494728 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.494730    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.494929 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.494752    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb16b12-9033-4c77-911e-0da2080390e1-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.494929 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.494842    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.494929 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.494863    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb16b12-9033-4c77-911e-0da2080390e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.494929 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.494885    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb16b12-9033-4c77-911e-0da2080390e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.495062 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.494955    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.495062 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.494981    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.495062 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.494998    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xpcv\" (UniqueName: \"kubernetes.io/projected/5bb16b12-9033-4c77-911e-0da2080390e1-kube-api-access-8xpcv\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.595739 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.595709    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.595888 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.595748    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.595888 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.595772    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xpcv\" (UniqueName: \"kubernetes.io/projected/5bb16b12-9033-4c77-911e-0da2080390e1-kube-api-access-8xpcv\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.595888 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.595816    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.595888 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.595835    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.596096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.595977    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb16b12-9033-4c77-911e-0da2080390e1-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.596096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.596033    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.596096 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.596065    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb16b12-9033-4c77-911e-0da2080390e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.596248 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.596096    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb16b12-9033-4c77-911e-0da2080390e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.596248 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.596117    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.596248 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.596213    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.596441 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.596294    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.596502 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.596438    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.596763 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.596736    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb16b12-9033-4c77-911e-0da2080390e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.598146 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.598121    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb16b12-9033-4c77-911e-0da2080390e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.598709 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.598689    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb16b12-9033-4c77-911e-0da2080390e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.608064 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.608039    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb16b12-9033-4c77-911e-0da2080390e1-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.608302 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.608285    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xpcv\" (UniqueName: \"kubernetes.io/projected/5bb16b12-9033-4c77-911e-0da2080390e1-kube-api-access-8xpcv\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-2gmnx\" (UID: \"5bb16b12-9033-4c77-911e-0da2080390e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.697920 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.697848    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:11.977421 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.977396    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"]
Nov 02 10:33:11.979572 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:33:11.979545    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb16b12_9033_4c77_911e_0da2080390e1.slice/crio-30d7d18096b111b8dc3b1a5699128e85e2bcbdee8e3f3b7f6187f9f1705c4935 WatchSource:0}: Error finding container 30d7d18096b111b8dc3b1a5699128e85e2bcbdee8e3f3b7f6187f9f1705c4935: Status 404 returned error can't find the container with id 30d7d18096b111b8dc3b1a5699128e85e2bcbdee8e3f3b7f6187f9f1705c4935
Nov 02 10:33:11.981552 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.981525    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:33:11.981610 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.981583    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:33:11.981650 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:11.981613    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:33:12.344657 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:12.344567    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx" event={"ID":"5bb16b12-9033-4c77-911e-0da2080390e1","Type":"ContainerStarted","Data":"cecd497ddeb7fb58a8d7fd914753a708cc23000b2d699ff0b1ab73bd002e02ac"}
Nov 02 10:33:12.344657 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:12.344630    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx" event={"ID":"5bb16b12-9033-4c77-911e-0da2080390e1","Type":"ContainerStarted","Data":"30d7d18096b111b8dc3b1a5699128e85e2bcbdee8e3f3b7f6187f9f1705c4935"}
Nov 02 10:33:12.371374 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:12.370314    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx" podStartSLOduration=1.370293257 podStartE2EDuration="1.370293257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:33:12.368863889 +0000 UTC m=+3473.201521973" watchObservedRunningTime="2026-11-02 10:33:12.370293257 +0000 UTC m=+3473.202951339"
Nov 02 10:33:12.698567 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:12.698478    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:13.699698 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:13.699662    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 10:33:13.700111 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:13.699715    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx" podUID="5bb16b12-9033-4c77-911e-0da2080390e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 10:33:14.699258 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:14.699221    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 10:33:14.699468 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:14.699279    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx" podUID="5bb16b12-9033-4c77-911e-0da2080390e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 10:33:15.699168 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:15.699130    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 10:33:15.699614 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:15.699207    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx" podUID="5bb16b12-9033-4c77-911e-0da2080390e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 10:33:16.699108 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:16.699076    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 10:33:16.699288 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:16.699139    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx" podUID="5bb16b12-9033-4c77-911e-0da2080390e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 10:33:17.698680 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:17.698648    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 10:33:17.698862 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:17.698710    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx" podUID="5bb16b12-9033-4c77-911e-0da2080390e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 10:33:18.698770 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:18.698732    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 10:33:18.699166 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:18.698799    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx" podUID="5bb16b12-9033-4c77-911e-0da2080390e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 10:33:19.699027 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:19.698987    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 10:33:19.699453 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:19.699056    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx" podUID="5bb16b12-9033-4c77-911e-0da2080390e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 10:33:21.222819 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:21.222771    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:21.223324 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:21.223123    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:21.223940 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:21.223919    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx"
Nov 02 10:33:26.404239 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.404205    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"]
Nov 02 10:33:26.407875 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.407855    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.422249 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.422222    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"]
Nov 02 10:33:26.547665 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.547630    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.547832 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.547671    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.547832 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.547691    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.547832 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.547750    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.547984 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.547854    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.547984 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.547895    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.547984 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.547913    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.547984 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.547935    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjpbx\" (UniqueName: \"kubernetes.io/projected/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-kube-api-access-xjpbx\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.547984 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.547952    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.649011 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.648977    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.649011 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.649016    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.649274 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.649036    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjpbx\" (UniqueName: \"kubernetes.io/projected/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-kube-api-access-xjpbx\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.649274 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.649060    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.649274 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.649100    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.649274 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.649131    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.649274 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.649157    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.649553 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.649326    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.649553 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.649419    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.649666 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.649630    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.649727 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.649703    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.649787 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.649752    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.649835 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.649787    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.650551 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.650525    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.651482 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.651457    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.651586 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.651511    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.660941 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.660888    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.661148 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.661125    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjpbx\" (UniqueName: \"kubernetes.io/projected/629f8a0c-a44b-4c35-aeb9-aaa31d82043e-kube-api-access-xjpbx\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-cmgx5\" (UID: \"629f8a0c-a44b-4c35-aeb9-aaa31d82043e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:26.721856 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:26.721832    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:27.000847 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:27.000820    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"]
Nov 02 10:33:27.002938 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:33:27.002908    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629f8a0c_a44b_4c35_aeb9_aaa31d82043e.slice/crio-535124d2c8d515e2e10bb73a27114ff83b85ea3076f4650a844c9a1aed08edae WatchSource:0}: Error finding container 535124d2c8d515e2e10bb73a27114ff83b85ea3076f4650a844c9a1aed08edae: Status 404 returned error can't find the container with id 535124d2c8d515e2e10bb73a27114ff83b85ea3076f4650a844c9a1aed08edae
Nov 02 10:33:27.005109 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:27.005079    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:33:27.005188 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:27.005141    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:33:27.005188 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:27.005170    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:33:27.424878 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:27.424789    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5" event={"ID":"629f8a0c-a44b-4c35-aeb9-aaa31d82043e","Type":"ContainerStarted","Data":"3e1beda37572605dc0b8ebfe6ebfdec9bbaa3a0dee6bbc040d3d2e12d61fe021"}
Nov 02 10:33:27.424878 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:27.424850    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5" event={"ID":"629f8a0c-a44b-4c35-aeb9-aaa31d82043e","Type":"ContainerStarted","Data":"535124d2c8d515e2e10bb73a27114ff83b85ea3076f4650a844c9a1aed08edae"}
Nov 02 10:33:27.461578 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:27.461531    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5" podStartSLOduration=1.461517733 podStartE2EDuration="1.461517733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:33:27.459245899 +0000 UTC m=+3488.291903980" watchObservedRunningTime="2026-11-02 10:33:27.461517733 +0000 UTC m=+3488.294175812"
Nov 02 10:33:27.722785 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:27.722756    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:28.723794 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:28.723757    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 10:33:28.724178 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:28.723807    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5" podUID="629f8a0c-a44b-4c35-aeb9-aaa31d82043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 10:33:29.722568 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:29.722527    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 10:33:29.722747 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:29.722587    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5" podUID="629f8a0c-a44b-4c35-aeb9-aaa31d82043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 10:33:30.723035 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:30.722996    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 10:33:30.723395 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:30.723061    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5" podUID="629f8a0c-a44b-4c35-aeb9-aaa31d82043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 10:33:31.722246 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:31.722210    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 10:33:31.722459 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:31.722261    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5" podUID="629f8a0c-a44b-4c35-aeb9-aaa31d82043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 10:33:32.722363 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:32.722324    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 10:33:32.722738 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:32.722403    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5" podUID="629f8a0c-a44b-4c35-aeb9-aaa31d82043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 10:33:33.722732 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:33.722696    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 10:33:33.723122 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:33.722748    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5" podUID="629f8a0c-a44b-4c35-aeb9-aaa31d82043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 10:33:34.722934 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:34.722898    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 02 10:33:34.723406 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:34.722961    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5" podUID="629f8a0c-a44b-4c35-aeb9-aaa31d82043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 02 10:33:36.309982 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:36.309938    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:36.310447 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:36.310346    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:36.311278 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:33:36.311259    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5"
Nov 02 10:33:39.635563 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:33:39.635526    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:34:09.636031 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:34:09.635993    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:34:39.635260 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:34:39.635216    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:35:09.635226 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:35:09.635185    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:35:11.544308 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.544272    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"]
Nov 02 10:35:11.550750 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.549964    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.563044 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.563016    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"]
Nov 02 10:35:11.644759 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.644725    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.644929 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.644767    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.644929 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.644896    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715f8b03-99be-4bbc-8817-edfbd731f24c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.645041 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.644946    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.645041 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.644987    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715f8b03-99be-4bbc-8817-edfbd731f24c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.645151 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.645103    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.645223 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.645174    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715f8b03-99be-4bbc-8817-edfbd731f24c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.645285 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.645217    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmcmw\" (UniqueName: \"kubernetes.io/projected/715f8b03-99be-4bbc-8817-edfbd731f24c-kube-api-access-tmcmw\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.645285 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.645255    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.746490 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.746453    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715f8b03-99be-4bbc-8817-edfbd731f24c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.746689 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.746497    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.746689 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.746537    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715f8b03-99be-4bbc-8817-edfbd731f24c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.746689 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.746637    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.746878 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.746698    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715f8b03-99be-4bbc-8817-edfbd731f24c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.746878 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.746743    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmcmw\" (UniqueName: \"kubernetes.io/projected/715f8b03-99be-4bbc-8817-edfbd731f24c-kube-api-access-tmcmw\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.746878 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.746770    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.746878 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.746819    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.746878 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.746850    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.747123 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.746921    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.747123 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.746998    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.747221 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.747148    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.747221 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.747198    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.747476 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.747459    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/715f8b03-99be-4bbc-8817-edfbd731f24c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.748995 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.748971    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/715f8b03-99be-4bbc-8817-edfbd731f24c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.749137 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.749118    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/715f8b03-99be-4bbc-8817-edfbd731f24c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.758788 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.758767    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmcmw\" (UniqueName: \"kubernetes.io/projected/715f8b03-99be-4bbc-8817-edfbd731f24c-kube-api-access-tmcmw\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.758893 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.758871    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/715f8b03-99be-4bbc-8817-edfbd731f24c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-sqhmk\" (UID: \"715f8b03-99be-4bbc-8817-edfbd731f24c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:11.866761 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:11.866676    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:12.146061 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:12.146035    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"]
Nov 02 10:35:12.147790 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:35:12.147767    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715f8b03_99be_4bbc_8817_edfbd731f24c.slice/crio-bcd06e02307672e339a5d1f154f5f3a9386d742d5ebf3dbc0090ce3e2a07f7db WatchSource:0}: Error finding container bcd06e02307672e339a5d1f154f5f3a9386d742d5ebf3dbc0090ce3e2a07f7db: Status 404 returned error can't find the container with id bcd06e02307672e339a5d1f154f5f3a9386d742d5ebf3dbc0090ce3e2a07f7db
Nov 02 10:35:12.149807 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:12.149767    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:35:12.149897 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:12.149839    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:35:12.149897 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:12.149868    4614 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 10:35:12.909513 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:12.909477    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk" event={"ID":"715f8b03-99be-4bbc-8817-edfbd731f24c","Type":"ContainerStarted","Data":"b481fb92f0c063ef126f133d652f69ff40d5065ace1beaad3547ea608cefb85d"}
Nov 02 10:35:12.910032 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:12.909528    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk" event={"ID":"715f8b03-99be-4bbc-8817-edfbd731f24c","Type":"ContainerStarted","Data":"bcd06e02307672e339a5d1f154f5f3a9386d742d5ebf3dbc0090ce3e2a07f7db"}
Nov 02 10:35:13.867113 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:13.867065    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:13.868584 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:13.868556    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 10:35:13.868725 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:13.868611    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk" podUID="715f8b03-99be-4bbc-8817-edfbd731f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 10:35:14.867934 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:14.867893    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 10:35:14.868407 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:14.867952    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk" podUID="715f8b03-99be-4bbc-8817-edfbd731f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 10:35:15.868001 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:15.867958    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 10:35:15.868474 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:15.868017    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk" podUID="715f8b03-99be-4bbc-8817-edfbd731f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 10:35:16.867555 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:16.867519    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 10:35:16.867747 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:16.867576    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk" podUID="715f8b03-99be-4bbc-8817-edfbd731f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 10:35:17.868087 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:17.868050    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 10:35:17.868539 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:17.868101    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk" podUID="715f8b03-99be-4bbc-8817-edfbd731f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 10:35:18.867964 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:18.867924    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 10:35:18.868159 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:18.867980    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk" podUID="715f8b03-99be-4bbc-8817-edfbd731f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 10:35:19.867381 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:19.867337    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 02 10:35:19.867574 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:19.867400    4614 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk" podUID="715f8b03-99be-4bbc-8817-edfbd731f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 02 10:35:20.296452 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:20.296421    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:35:20.334425 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:20.334399    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:35:21.315199 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:21.315168    4614 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:21.315713 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:21.315579    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:21.316644 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:21.316625    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk"
Nov 02 10:35:21.339462 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:35:21.339409    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk" podStartSLOduration=10.339393585 podStartE2EDuration="10.339393585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 10:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:35:12.942715797 +0000 UTC m=+3593.775373878" watchObservedRunningTime="2026-11-02 10:35:21.339393585 +0000 UTC m=+3602.172051666"
Nov 02 10:35:39.635120 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:35:39.635084    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:36:09.634957 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:36:09.634881    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:36:39.634595 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:36:39.634554    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:37:09.635206 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:37:09.635167    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:37:39.635340 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:37:39.635302    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:38:09.635238 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:38:09.635200    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:38:39.634654 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:38:39.634611    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:39:09.635607 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:39:09.635536    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:39:39.634986 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:39:39.634949    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:40:07.267500 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:40:07.267455    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:40:07.268277 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:40:07.268239    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx" podUID="5bb16b12-9033-4c77-911e-0da2080390e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:40:07.354375 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:40:07.353496    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:40:07.358373 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:40:07.354918    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5" podUID="629f8a0c-a44b-4c35-aeb9-aaa31d82043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:40:07.359465 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:40:07.359432    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:40:07.359677 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:40:07.359649    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk" podUID="715f8b03-99be-4bbc-8817-edfbd731f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:40:09.635479 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:40:09.635439    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:40:20.324413 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:40:20.324378    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:40:20.364833 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:40:20.364806    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:40:37.265513 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:40:37.265463    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:40:37.266274 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:40:37.266237    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-2gmnx" podUID="5bb16b12-9033-4c77-911e-0da2080390e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:40:37.361375 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:40:37.359470    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:40:37.361375 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:40:37.359555    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-cmgx5" podUID="629f8a0c-a44b-4c35-aeb9-aaa31d82043e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:40:37.372375 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:40:37.370455    4614 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 10:40:37.372375 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:40:37.370564    4614 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-sqhmk" podUID="715f8b03-99be-4bbc-8817-edfbd731f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 10:40:39.634503 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:40:39.634467    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:41:09.634374 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:41:09.634317    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:41:39.635116 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:41:39.635078    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:42:09.635084 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:42:09.635008    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:42:39.635106 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:42:39.635067    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:43:09.635344 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:43:09.635302    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:43:12.012100 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:12.012071    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cfpwn_b0b53ec3-a9e4-45d7-8aa4-59237abc2614/manager/0.log"
Nov 02 10:43:12.415944 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:12.415869    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-j8qxd_7f946e8c-3ccb-4967-9d2d-61f67abb41ed/manager/0.log"
Nov 02 10:43:14.250513 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:14.250482    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6647fcb89f-kzxjl_0d5bf9bb-32fa-4d3e-9f35-b050565e4782/maas-api/0.log"
Nov 02 10:43:18.135752 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:18.135726    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-sd2jc_c0cc5dd2-053c-4a5b-8c6b-001ea092716e/postgres/0.log"
Nov 02 10:43:22.387654 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:22.387622    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6p6b2_231615b6-94b8-440a-97ae-dcdca4860543/discovery/0.log"
Nov 02 10:43:22.578095 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:22.578069    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c759d848c-77mmj_b215a15b-8411-4d37-af02-20819dcb01c9/kube-auth-proxy/0.log"
Nov 02 10:43:22.933226 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:22.933197    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-2gmnx_5bb16b12-9033-4c77-911e-0da2080390e1/istio-proxy/0.log"
Nov 02 10:43:23.486643 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:23.486611    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-cmgx5_629f8a0c-a44b-4c35-aeb9-aaa31d82043e/istio-proxy/0.log"
Nov 02 10:43:24.549661 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:24.549627    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-sqhmk_715f8b03-99be-4bbc-8817-edfbd731f24c/istio-proxy/0.log"
Nov 02 10:43:25.258683 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:25.258654    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d88c448f-z5k5p_80746de3-f9f3-4efe-a963-9b37f290fcb7/router/0.log"
Nov 02 10:43:39.635664 ip-10-0-134-174 kubenswrapper[4614]: E1102 10:43:39.635624    4614 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nbfgg" containerName="extractor"
Nov 02 10:43:41.714274 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:41.714234    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lxn8z_c5b95e7b-2ce0-45a1-9859-0d2378576064/global-pull-secret-syncer/0.log"
Nov 02 10:43:41.828132 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:41.828105    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-cf7t5_78cde7e0-a2f3-4f33-ba88-6c9fedc723eb/connection-checker/0.log"
Nov 02 10:43:41.858101 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:41.858078    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-pprd9_95d7a9df-37e3-492b-9540-3df603931723/connection-checker/0.log"
Nov 02 10:43:41.888717 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:41.888692    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xtlc6_fce2663d-12a8-427f-b8ec-0d2238a06470/connection-checker/0.log"
Nov 02 10:43:42.041145 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:42.041115    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xzrll_e8df50bb-8d26-42c1-b496-c9d9f56785d9/konnectivity-agent/0.log"
Nov 02 10:43:42.104028 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:42.103997    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-174.us-east-2.compute.internal_a2d480d50be16b70f16fe76e324fa19d/haproxy/0.log"
Nov 02 10:43:45.426625 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:45.426597    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cfpwn_b0b53ec3-a9e4-45d7-8aa4-59237abc2614/manager/0.log"
Nov 02 10:43:45.903477 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:45.903446    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-j8qxd_7f946e8c-3ccb-4967-9d2d-61f67abb41ed/manager/0.log"
Nov 02 10:43:47.541858 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:47.541826    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cfpwn_b0b53ec3-a9e4-45d7-8aa4-59237abc2614/manager/0.log"
Nov 02 10:43:47.982164 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:47.982123    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-j8qxd_7f946e8c-3ccb-4967-9d2d-61f67abb41ed/manager/0.log"
Nov 02 10:43:49.911176 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:49.911140    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cfpwn_b0b53ec3-a9e4-45d7-8aa4-59237abc2614/manager/0.log"
Nov 02 10:43:50.329839 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:50.329808    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-j8qxd_7f946e8c-3ccb-4967-9d2d-61f67abb41ed/manager/0.log"
Nov 02 10:43:51.598973 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:51.598946    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-mlxbp_3c01c082-8479-460d-a80d-5e590ab2fcbe/cluster-monitoring-operator/0.log"
Nov 02 10:43:51.991731 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:51.991704    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nd59c_44a54ffa-79c6-4dfb-9acc-f045c5538d1f/node-exporter/0.log"
Nov 02 10:43:52.020493 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.020472    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nd59c_44a54ffa-79c6-4dfb-9acc-f045c5538d1f/kube-rbac-proxy/0.log"
Nov 02 10:43:52.045968 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.045946    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nd59c_44a54ffa-79c6-4dfb-9acc-f045c5538d1f/init-textfile/0.log"
Nov 02 10:43:52.240505 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.240467    4614 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"]
Nov 02 10:43:52.244158 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.244096    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.252486 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.252460    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"]
Nov 02 10:43:52.253454 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.253432    4614 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zwdzc\"/\"default-dockercfg-ktm6x\""
Nov 02 10:43:52.253566 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.253502    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zwdzc\"/\"openshift-service-ca.crt\""
Nov 02 10:43:52.253634 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.253578    4614 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zwdzc\"/\"kube-root-ca.crt\""
Nov 02 10:43:52.319106 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.319063    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-podres\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.319290 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.319174    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-sys\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.319290 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.319276    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-lib-modules\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.319456 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.319328    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-proc\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.319728 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.319449    4614 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff9nd\" (UniqueName: \"kubernetes.io/projected/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-kube-api-access-ff9nd\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.419949 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.419912    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff9nd\" (UniqueName: \"kubernetes.io/projected/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-kube-api-access-ff9nd\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.420142 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.419979    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-podres\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.420142 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.420027    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-sys\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.420142 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.420064    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-lib-modules\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.420142 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.420112    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-sys\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.420142 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.420112    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-podres\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.420377 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.420222    4614 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-proc\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.420377 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.420245    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-lib-modules\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.420377 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.420303    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-proc\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.432270 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.432246    4614 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff9nd\" (UniqueName: \"kubernetes.io/projected/0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb-kube-api-access-ff9nd\") pod \"perf-node-gather-daemonset-wp4ll\" (UID: \"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb\") " pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.556337 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.556248    4614 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:52.845864 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:52.845825    4614 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"]
Nov 02 10:43:52.847927 ip-10-0-134-174 kubenswrapper[4614]: W1102 10:43:52.847900    4614 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0baa0587_2a0e_4df2_bbb7_8f4087fe8dfb.slice/crio-66c986d6bd651e4121963e7384278ffe16f890d4bbe2b47b1df86cabf0de2a0c WatchSource:0}: Error finding container 66c986d6bd651e4121963e7384278ffe16f890d4bbe2b47b1df86cabf0de2a0c: Status 404 returned error can't find the container with id 66c986d6bd651e4121963e7384278ffe16f890d4bbe2b47b1df86cabf0de2a0c
Nov 02 10:43:53.097348 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:53.097264    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6p6b2_231615b6-94b8-440a-97ae-dcdca4860543/discovery/0.log"
Nov 02 10:43:53.107953 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:53.107928    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c759d848c-77mmj_b215a15b-8411-4d37-af02-20819dcb01c9/kube-auth-proxy/0.log"
Nov 02 10:43:53.143511 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:53.143482    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-2gmnx_5bb16b12-9033-4c77-911e-0da2080390e1/istio-proxy/0.log"
Nov 02 10:43:53.207425 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:53.207366    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-cmgx5_629f8a0c-a44b-4c35-aeb9-aaa31d82043e/istio-proxy/0.log"
Nov 02 10:43:53.320690 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:53.320662    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-sqhmk_715f8b03-99be-4bbc-8817-edfbd731f24c/istio-proxy/0.log"
Nov 02 10:43:53.395858 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:53.395787    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d88c448f-z5k5p_80746de3-f9f3-4efe-a963-9b37f290fcb7/router/0.log"
Nov 02 10:43:53.857286 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:53.857254    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll" event={"ID":"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb","Type":"ContainerStarted","Data":"859fa14c3820ab6afc0597e7966866076a20e7e8958c126b9f113556ec1a6da4"}
Nov 02 10:43:53.857696 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:53.857303    4614 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll" event={"ID":"0baa0587-2a0e-4df2-bbb7-8f4087fe8dfb","Type":"ContainerStarted","Data":"66c986d6bd651e4121963e7384278ffe16f890d4bbe2b47b1df86cabf0de2a0c"}
Nov 02 10:43:53.857696 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:53.857379    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:43:53.893651 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:53.893609    4614 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll" podStartSLOduration=1.893596367 podStartE2EDuration="1.893596367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 10:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 10:43:53.892515291 +0000 UTC m=+4114.725173371" watchObservedRunningTime="2026-11-02 10:43:53.893596367 +0000 UTC m=+4114.726254441"
Nov 02 10:43:54.000065 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:54.000039    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-qrb6h_e5393080-e20a-47fc-8dc7-125f42457736/networking-console-plugin/0.log"
Nov 02 10:43:54.528011 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:54.527979    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/3.log"
Nov 02 10:43:54.534094 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:54.534055    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n99gd_2d28887e-4aa9-4209-a51a-fec831136660/console-operator/4.log"
Nov 02 10:43:55.698273 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:55.698222    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-sb5ns_b9ed0d41-1122-4ec2-a9af-e84cc71605df/volume-data-source-validator/0.log"
Nov 02 10:43:57.183491 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:57.183462    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zgjvs_502f9765-e116-4359-b513-c1d00c90d7de/dns/0.log"
Nov 02 10:43:57.209450 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:57.209419    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zgjvs_502f9765-e116-4359-b513-c1d00c90d7de/kube-rbac-proxy/0.log"
Nov 02 10:43:57.300111 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:57.300078    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zz4q2_92f5b78b-58a0-4d0d-9bf6-97c4943597fd/dns-node-resolver/0.log"
Nov 02 10:43:57.966723 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:57.966696    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xz8m5_77530df7-545e-42b0-ad16-ce934cb10dab/node-ca/0.log"
Nov 02 10:43:59.795826 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:59.795799    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6p6b2_231615b6-94b8-440a-97ae-dcdca4860543/discovery/0.log"
Nov 02 10:43:59.872543 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:43:59.872514    4614 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zwdzc/perf-node-gather-daemonset-wp4ll"
Nov 02 10:44:00.131515 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:00.131441    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c759d848c-77mmj_b215a15b-8411-4d37-af02-20819dcb01c9/kube-auth-proxy/0.log"
Nov 02 10:44:00.196733 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:00.196711    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-2gmnx_5bb16b12-9033-4c77-911e-0da2080390e1/istio-proxy/0.log"
Nov 02 10:44:00.293680 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:00.293650    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-cmgx5_629f8a0c-a44b-4c35-aeb9-aaa31d82043e/istio-proxy/0.log"
Nov 02 10:44:00.495454 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:00.495426    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-sqhmk_715f8b03-99be-4bbc-8817-edfbd731f24c/istio-proxy/0.log"
Nov 02 10:44:00.629092 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:00.629064    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d88c448f-z5k5p_80746de3-f9f3-4efe-a963-9b37f290fcb7/router/0.log"
Nov 02 10:44:02.679937 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:02.679908    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xkrcs_26adcc96-ef0b-416f-8b15-e01be59fb5ae/serve-healthcheck-canary/0.log"
Nov 02 10:44:03.316646 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:03.316614    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-pdw8n_85741a00-b702-46cb-a90c-e000bf9976f0/insights-operator/0.log"
Nov 02 10:44:03.318557 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:03.318534    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-pdw8n_85741a00-b702-46cb-a90c-e000bf9976f0/insights-operator/1.log"
Nov 02 10:44:03.442160 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:03.442126    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nbfgg_b36bfac6-a600-4029-a446-7de82575fb0a/kube-rbac-proxy/0.log"
Nov 02 10:44:03.469312 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:03.469283    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6p6b2_231615b6-94b8-440a-97ae-dcdca4860543/discovery/0.log"
Nov 02 10:44:03.472885 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:03.472867    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nbfgg_b36bfac6-a600-4029-a446-7de82575fb0a/exporter/0.log"
Nov 02 10:44:03.484976 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:03.484952    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c759d848c-77mmj_b215a15b-8411-4d37-af02-20819dcb01c9/kube-auth-proxy/0.log"
Nov 02 10:44:03.515720 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:03.515697    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nbfgg_b36bfac6-a600-4029-a446-7de82575fb0a/extractor/0.log"
Nov 02 10:44:03.535125 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:03.535106    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-2gmnx_5bb16b12-9033-4c77-911e-0da2080390e1/istio-proxy/0.log"
Nov 02 10:44:03.592933 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:03.592877    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-cmgx5_629f8a0c-a44b-4c35-aeb9-aaa31d82043e/istio-proxy/0.log"
Nov 02 10:44:03.646706 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:03.646681    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-79lct-772qs_b0bfa1c8-8148-4bdb-abd6-cc24256fd371/insights-gathering/0.log"
Nov 02 10:44:03.735411 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:03.735383    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-sqhmk_715f8b03-99be-4bbc-8817-edfbd731f24c/istio-proxy/0.log"
Nov 02 10:44:03.826680 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:03.826653    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d88c448f-z5k5p_80746de3-f9f3-4efe-a963-9b37f290fcb7/router/0.log"
Nov 02 10:44:04.633855 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:04.633813    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-f7698d755-k98vj_25ea051f-bf6f-4b0e-ac1e-91ff591f659f/manager/0.log"
Nov 02 10:44:04.642038 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:04.642017    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-f7698d755-k98vj_25ea051f-bf6f-4b0e-ac1e-91ff591f659f/copy-manifests/0.log"
Nov 02 10:44:05.192364 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:05.192335    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cfpwn_b0b53ec3-a9e4-45d7-8aa4-59237abc2614/manager/0.log"
Nov 02 10:44:05.607209 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:05.607174    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-j8qxd_7f946e8c-3ccb-4967-9d2d-61f67abb41ed/manager/0.log"
Nov 02 10:44:06.398077 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:06.398045    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-f7698d755-k98vj_25ea051f-bf6f-4b0e-ac1e-91ff591f659f/manager/0.log"
Nov 02 10:44:06.422830 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:06.422806    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-f7698d755-k98vj_25ea051f-bf6f-4b0e-ac1e-91ff591f659f/copy-manifests/0.log"
Nov 02 10:44:07.251605 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:07.251569    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6p6b2_231615b6-94b8-440a-97ae-dcdca4860543/discovery/0.log"
Nov 02 10:44:07.285436 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:07.285408    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c759d848c-77mmj_b215a15b-8411-4d37-af02-20819dcb01c9/kube-auth-proxy/0.log"
Nov 02 10:44:07.328298 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:07.328263    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-2gmnx_5bb16b12-9033-4c77-911e-0da2080390e1/istio-proxy/0.log"
Nov 02 10:44:07.392283 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:07.392249    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-cmgx5_629f8a0c-a44b-4c35-aeb9-aaa31d82043e/istio-proxy/0.log"
Nov 02 10:44:07.523042 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:07.522965    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-sqhmk_715f8b03-99be-4bbc-8817-edfbd731f24c/istio-proxy/0.log"
Nov 02 10:44:07.612104 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:07.612073    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d88c448f-z5k5p_80746de3-f9f3-4efe-a963-9b37f290fcb7/router/0.log"
Nov 02 10:44:07.992200 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:07.992172    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6p6b2_231615b6-94b8-440a-97ae-dcdca4860543/discovery/0.log"
Nov 02 10:44:08.029578 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:08.029553    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c759d848c-77mmj_b215a15b-8411-4d37-af02-20819dcb01c9/kube-auth-proxy/0.log"
Nov 02 10:44:08.065460 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:08.065434    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-2gmnx_5bb16b12-9033-4c77-911e-0da2080390e1/istio-proxy/0.log"
Nov 02 10:44:08.118819 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:08.118794    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-cmgx5_629f8a0c-a44b-4c35-aeb9-aaa31d82043e/istio-proxy/0.log"
Nov 02 10:44:08.240820 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:08.240795    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-sqhmk_715f8b03-99be-4bbc-8817-edfbd731f24c/istio-proxy/0.log"
Nov 02 10:44:08.321808 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:08.321730    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d88c448f-z5k5p_80746de3-f9f3-4efe-a963-9b37f290fcb7/router/0.log"
Nov 02 10:44:08.759741 ip-10-0-134-174 kubenswrapper[4614]: I1102 10:44:08.759715    4614 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cfpwn_b0b53ec3-a9e4-45d7-8aa4-59237abc2614/manager/0.log"